PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates

Paiheng Xu, Jing Liu, Wei Ai

arXiv:2606.03029v1cs.CLcs.AI

TL;DR

Existing hypothesis-generation methods emphasize globally discriminative textual patterns, which may reflect confounds rather than substantive differences. This paper conditions discovery on researcher-specified strata, using interactions for sign reversal and demeaning with reweighting for imbalance. Synthetic experiments and expert evaluations indicate that the covariate-aware methods recover targeted patterns and produce more useful hypotheses in the tested settings.

  • Problem

    Global selection can mistake confounded patterns for substantive text–outcome differences because it ignores researcher-specified covariates.

  • Method

    The framework incorporates covariates through feature–covariate interactions for sign reversal and within-stratum demeaning with inverse-frequency reweighting for imbalance.

  • Results

    Synthetic experiments show each method performs best in its targeted setting, while expert evaluations on CONGRESS and NCTE favor covariate-aware hypotheses.

  • Takeaways & Limitations

    Covariate-aware feature selection steers discovery toward within-stratum differences and hypotheses that experts find more useful in the evaluated domains.

  • Takeaways & Limitations

    The framework depends on researcher-specified covariates and cannot discover relevant omitted covariates or make causal claims.

Abstract

from arXiv · show

A core goal of computational social science is to discover interpretable differences in how language varies across outcomes of interest, such as political affiliation or instructional quality. Recent LLM-based hypothesis generation methods describe such differences in natural language, but select for globally discriminative patterns without accounting for covariates that shape the data based on researchers' domain knowledge. When covariates are ignored, selected patterns can reflect confounds rather than differences of substantive interest. We introduce conditional hypothesis generation, a framework that incorporates researcher-specified covariates to steer hypothesis discovery toward differences that hold within relevant subgroups. Two challenges arise: the target subgroup may be underrepresented (stratum imbalance), and the direction of a difference may reverse across subgroups (sign reversal). We propose two econometrics-inspired methods: one introduces feature--covariate interactions to detect sign reversals, and the other applies within-stratum demeaning and inverse-frequency reweighting to equalize underrepresented strata. Synthetic experiments show each method outperforms global baselines in its targeted setting, and expert evaluation on two real-world datasets confirms that covariate-aware generation surfaces more useful hypotheses within relevant subgroups.

1 Introduction

Conditional hypothesis generation addresses the limits of globally discriminative text patterns by incorporating researcher-specified covariates. The paper proposes methods for stratum imbalance and sign reversal, with synthetic and expert evaluations supporting covariate-aware discovery.

  • LLM-based methods generate natural-language hypotheses from textual patterns but typically select them by global discrimination between outcome groups.
  • Global discrimination can capture confounds, motivating hypotheses that hold within researcher-specified covariate strata.The paper gives uneven national-park distribution across states as an example of a party-predictive but not inherently partisan feature.
  • Conditional hypothesis generation incorporates covariates such as policy area, time period, or classroom environment without requiring the hypothesis itself to be known in advance.
  • Conditioning creates two challenges: underrepresented strata can have their signals dominated, while sign reversal can cancel conditional differences during aggregation.
  • Interaction-lasso targets sign reversal, whereas demeaned-reweighted-lasso isolates within-stratum variation and reweights rare strata.
  • Synthetic evaluations show demeaned-reweighted-lasso outperforming global baselines under imbalance and interaction-lasso recovering differences under sign reversal.
  • Expert evaluations on CONGRESS and NCTE rate covariate-aware hypotheses as more useful than hypotheses unique to the global baseline.

2 Preliminaries

The task generates natural-language descriptions of how text varies with an outcome while allowing researchers to condition discovery on selected covariates. The paper builds on sample-and-propose LLM methods and SAE-based feature selection.

  • Hypothesis generation produces natural-language statements describing how a target variable varies with text content.The target may indicate group membership, and the output is a pre-specified set H of hypotheses.
  • Documents are associated with covariates such as time period, author demographics, or topic, from which researchers may select relevant strata.
  • Existing LLM-based approaches sample labeled examples and prompt an LLM to induce textual patterns and propose hypotheses.
  • Movva et al. select predictive features over documents with LASSO, then use an LLM to describe selected features from feature-related texts.
  • Sparse Autoencoders map document embeddings to sparse, interpretable features that support feature-based hypothesis selection.Each active feature is intended to capture a coherent concept.

3 Method

The method models conditional text–outcome differences using either feature–covariate interactions or within-stratum residualization with inverse-frequency weighting. The choice depends on whether covariates create sign reversal or primarily produce stratum imbalance.

  • Conditional hypothesis generation targets group differences that are discriminative within covariate strata, addressing stratum imbalance and sign reversal.
  • Baseline LASSO selects globally discriminative SAE features by ranking coefficients from an L1-regularized logistic regression.
  • Interaction-lasso: Interaction-lasso augments SAE features with covariates and feature–covariate products so coefficients can vary across strata.Features are ranked by the largest absolute main or interaction effect, while covariate main effects serve as nuisance controls.
  • Interaction-lasso: Interaction-lasso expands the feature space by M×P interaction features, increasing computational cost and potentially destabilizing selection in high dimensions.Rare covariates also make interaction terms nearly all-zero and their estimates noisy.
  • Demeaned-reweighted-lasso: Demeaning subtracts covariate-stratum means from SAE activations and outcomes to isolate within-stratum differences.The resulting coefficients capture feature-associated differences after partialling out stratum-level mean shifts.
  • Demeaned-reweighted-lasso: Demeaned-reweighted-lasso assigns inverse-frequency sample weights so underrepresented strata contribute comparably to feature selection.
  • Demeaning assumes a consistent difference sign across strata, whereas interaction-lasso is appropriate when the covariate changes that sign.

4 Experiments on Synthetic Datasets

Synthetic experiments test whether covariate-aware methods recover targeted hypotheses under stratum imbalance and sign reversal. Results show that method choice matters, with reweighting helping rare strata and interactions addressing reversed differences.

  • Experimental design: The experiments construct synthetic corpora from congressional bill summaries with controlled covariate structures for stratum imbalance and sign reversal.Topics, subtopics, and bill creation periods provide the corpus structure and ground-truth contrasts.
  • Scenario 1: Stratum imbalance: In the imbalance scenario, social-policy topics form the targeted strata while an imbalance ratio controls their frequency relative to government-and-economy topics.Lower r suppresses the targeted social-policy signal in one group, making recovery harder.
  • Scenario 2: Sign reversal: In the sign-reversal scenario, subtopic assignments switch between pre- and post-2011 periods, causing the targeted differences to reverse across strata.The setup evaluates recovery of five targeted subtopic differences and violates the consistent-direction assumption.
  • Results: Interaction-lasso is the only covariate-aware method that significantly improves over LASSO under sign reversal, increasing surface similarity from .484 to .600 and F1 from .362 to .576.The reported improvements have p=0.008 for surface similarity and p=0.002 for F1.
  • Results: LLM-direct methods fall behind SAE-based methods, and llm-covariate does not consistently outperform llm-global across sample sizes or with gpt-5.4.The comparison tests whether covariate-aware discovery can be obtained through prompting without corpus-level feature selection.

5 Validation on Real-World Datasets

The real-world evaluation compares covariate-aware and global-baseline hypotheses on CONGRESS and NCTE, using expert ratings of helpfulness and conditional interpretive value. Covariate-aware selection surfaces more instruction- or policy-oriented hypotheses, but the value of the specified covariate differs by dataset.

  • Study setup: The evaluation uses CONGRESS and NCTE, with domain scholars rating hypotheses unique to each method after semantic matching.CONGRESS uses policy-versus-procedure as its covariate, while NCTE uses behavioral management to examine instructional activities.
  • Study setup: Experts rate hypothesis helpfulness on a 1–5 scale and conditional interpretive value on a separate 1–5 scale.The evaluation reports hypotheses’ prevalence in the two outcome groups and limits each set to |H|=10.
  • Results: In NCTE, covariate-aware hypotheses shift from classroom-management context toward remediation and instructional activities such as individualized follow-up and peer explanation.The covariate-aware hypotheses are described as more aligned with the researcher-specified stratum than the global-baseline hypotheses.
  • Results: 3.10 vs. 2.50, averaged helpfulness ratings favored covariate-aware hypotheses over their global-baseline counterparts.The comparison is reported for hypotheses unique to each method after semantic matching.
  • Results: 3.50 vs. 2.33, NCTE’s conditional interpretive-value ratings favored unique covariate-aware hypotheses over unique LASSO hypotheses.In CONGRESS, covariate-aware hypotheses were more helpful, 3.25 vs. 2.75, but had lower conditional interpretive value, 1.50 vs. 2.50.

6 Related Work

The paper situates LLM-based hypothesis generation alongside classical text-analysis methods and covariate-adjustment techniques. Its distinction is to use researcher-specified covariate structure to steer discovery toward conditional differences of interest.

  • LLM-based hypothesis generation: Classical text-analysis approaches include n-gram frequency comparisons and topic models such as LDA and structural topic models.These methods relate text to target variables and remain widely used in computational social science.
  • Positioning: This work differs by incorporating domain knowledge through statistical models of covariate structure to steer discovery toward researcher-specified differences of interest.The covariates specify which conditions should shape hypothesis discovery.
  • Statistical methods for covariate adjustment: Covariate adjustment draws on fixed effects, within-group residualization, and the Frisch–Waugh–Lovell theorem to remove nuisance variation or make comparisons conditional on covariates.The paper connects these tools to Simpson’s paradox, where marginal associations can reverse relative to conditional patterns.

7 Conclusion

The paper introduces conditional hypothesis generation to steer natural-language discovery toward differences within researcher-specified strata. Its two methods target distinct statistical challenges, and both synthetic and expert evaluations support covariate-aware selection within the tested settings.

  • Framework: Conditional hypothesis generation steers natural-language hypothesis discovery toward differences that hold within researcher-specified covariate strata.Covariates let researchers encode domain knowledge about which strata should shape discovery.
  • Statistical challenges: The framework addresses stratum imbalance, which suppresses underrepresented-stratum signals, and sign reversal, where aggregation can cancel or misrepresent conditional differences.These challenges motivate distinct covariate-aware modeling choices.
  • Methods: Interaction-lasso models feature–covariate interactions for sign reversal, whereas demeaned-reweighted-lasso removes stratum-level variation and reweights underrepresented strata under a consistent-direction assumption.The methods are complementary rather than interchangeable across the two statistical settings.
  • Evidence: Synthetic experiments show demeaned-reweighted-lasso recovers hypotheses suppressed by stratum imbalance, while interaction-lasso is needed when difference directions reverse.Expert validation on CONGRESS and NCTE further suggests that covariate-aware methods surface more useful hypotheses than those unique to a global baseline.

Limitations

The framework depends on researcher-specified covariates and categorical strata, while inheriting limitations from the underlying SAE interpretation pipeline and synthetic and expert-based evaluation.

  • Covariate dependence: Researchers must choose covariates, so the methods cannot discover omitted covariates or make causal claims.Poorly chosen or noisy covariates may steer discovery toward unhelpful strata.
  • Covariate dependence: The methods focus on binary or categorical covariates, while continuous covariates require discretization or a different residualization strategy.Conditional hypotheses also depend on valid covariate operationalization, including automated annotations.
  • Method-specific assumptions: Demeaned-reweighted-lasso assumes consistent difference directions across strata and cannot recover stratum-specific effects under sign reversal.Interaction-lasso is better suited to sign reversal but expands the feature space.
  • Pipeline dependence: The implementation inherits SAE pipeline requirements involving full-corpus feature extraction, interpretable features, hyperparameters, and imperfect LLM verbalization.The contribution changes feature selection, not representation learning or natural-language generation.
  • Evaluation scope: Synthetic ground truth may not span applied linguistic phenomena, while real-world validation covers only two datasets and expert panels.Ratings indicate that covariate-aware selection can surface useful hypotheses, not a complete cross-domain benchmark.

A Synthetic Dataset Details

The synthetic data use eligible BILLS topics organized into Government & Economy and Social Policy pools, with three topics sampled from each pool per seed run.

  • Topic pools: The BILLS dataset provides 21 high-level topics and 114 low-level subtopics for synthetic corpus construction.Topics are filtered for sufficient subtopic sample sizes before scenario sampling.
  • Topic pools: Table 5 lists eligible topic pools for conditional discovery, with three topics randomly sampled from each pool in every seed run.The pools are Tgov for Government & Economy and Tsoc for Social Policy.

B Evaluation Metrics

Synthetic evaluation uses repeated train-validation-test splits and compares generated hypotheses with reference annotations through correlation-based matching and LLM-judged similarity.

  • Evaluation protocol: The experiments repeat each scenario across 10 random seeds and report each metric’s mean and standard deviation.Datasets are split into training, validation, and test sets in a 60:20:20 ratio.
  • Evaluation protocol: Reference annotations come directly from synthetic construction, while an LLM produces binary annotations for each generated hypothesis on the same test texts.Qwen3-30B-A3B-Instruct-2507 is used for hypothesis annotation.
  • Matching: Generated and reference annotation vectors form a correlation matrix, which the Hungarian algorithm converts into maximum-correlation one-to-one matches.The LLM used for judging surface similarity is gpt-4.1-2025-04-14.

C Method Implementation Details

The implementation compares global and direct-prompting baselines with covariate-aware selection, calibrates against oracle and random references, and tunes SAE configurations on validation data.

  • Baselines: SAE-lasso and SAE-separation score provide global feature-selection baselines using the same SAE activation matrix as the proposed methods.SAE-lasso selects the top |H| features with L1-regularized selection, while SAE-separation ranks prevalence differences between highly and non-activated texts.
  • Baselines: LLM-direct baselines compare prompts without covariates against prompts that expose covariate-profile labels and request within-stratum patterns.The covariate prompt prefixes each text with its covariate values.
  • LLM-direct settings: The prompt sweep tests 25, 50, and 100 sampled texts per corpus with gpt-4.1 and gpt-5.4.Because gains were inconsistent, the study fixes 100 texts and gpt-4.1 for the main LLM-direct baseline.
  • Reference baselines: Oracle supplies examples from exact positive and negative ground-truth sides, whereas Random measures recovery from arbitrary corpus splits and generic topical cues.Both references calibrate the synthetic interpretation evaluation.
  • SAE tuning: SAE configurations are tuned over feature counts M and sparsity K, with dataset-specific grids and validation-based selection.Synthetic configurations maximize validation surface similarity, while real-world configurations use predictive scores.
  • Expert-evaluation tables: Table 6 reports NCTE hypotheses with matched and unique sections, using colors for REMED direction and Δ = H/M − Low.The table’s color coding distinguishes increased mid/high REMED from increased low REMED.
  • Expert-evaluation tables: Table 7 reports Congress hypotheses with matched and unique sections, using red and blue for Republican and Democrat direction and Δ = Rep. − Dem.Alternating light shades improve row readability.

D.2 Covariate operationalization

The paper operationalizes covariates using dataset-specific labels and applies structured prompts to generate objective features from positive and negative samples. It also documents evaluation procedures, dataset splits, and prompt-budget sensitivity analyses.

  • CONGRESS uses a binary LLM annotation indicating whether a speech segment contains substantive public-policy discussion rather than only congressional procedure.
  • NCTE defines behavior-management strata using CLBM<6 for low-quality and CLBM≥6 for high-quality behavior management.The paper expects less management language in high-quality segments.
  • The SAE interpretation prompt asks for one objective, specific feature present in positive samples and absent from negative samples.The prompt explicitly rejects generic features that also appear in negative samples.
  • The NCTE dataset contains around 10K samples, split into 60:20:20 train, validation, and test sets, with approximately 5K mid/high-quality and 5K low-quality samples.Scores were provided by trained annotators and the covariates were pre-specified before expert annotation.
  • Expert annotators rated method-blinded hypotheses for helpfulness on a 1–5 scale and then assessed whether covariate-stratified statistics changed their ratings.The evaluation used manually curated one-to-one matched pairs and marked remaining hypotheses as method-unique.
  • Prompt-budget analyses varied sampled texts per corpus and found non-monotonic trends, with covariate-labeled prompting not consistently outperforming global prompting.The analyses were reported for LLM-direct baselines using gpt-4.1 and gpt-5.4; lines show seed means and shaded bands show ±1 standard deviation.
Loading 2606.03029v1…