PaperlayerView exact revision on arXiv

Source-linked AI summary

When AI Generates Covariates: Causal Typing and Estimand Drift in Sequential Experiments

Takes Fujita, Nobutaka Hattori

arXiv:2609.17772v1stat.MEcs.AIq-bio.QM

TL;DR

Unspecified roles for generated covariates can change the causal question in sequential experiments. The paper sets out causal typing and an estimand lock to govern confirmatory use; under audit correctness and standard identification assumptions, admissible assignments preserve the locked effect, while its discussion notes limitations including reliance on human judgment.

  • Problem

    Generated covariates may represent different causal roles, so their use can change the question unless their role is specified before analysis.

    Source: §1 · pp. 1–2

  • Method

    The paper combines versioned representations, causal-role and claim-status rules, and an estimand lock for confirmatory analyses.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Under audit correctness and standard identification assumptions, admissible role assignments identify the locked standardized proximal effect.

    Source: §14 · p. 23

  • Takeaways & Limitations

    Causal typing and the estimand lock constrain confirmatory use so generated representations do not silently change the target or claim.

    Source: §14 · p. 23

    Causal typing depends on human judgment, and the framework does not certify that the assigned roles are correct.

    Source: §13 · p. 23

Abstract

from arXiv · show

AI-generated covariates from notes, conversations, images, and wearable streams can change the causal question when their roles are left unspecified. A generated feature may represent a treatment version, pre-action state, history, design variable, mediator, outcome proxy, observation process, or intercurrent event; these roles are not interchangeable. We formulate a causal type discipline for sequential experiments: a versioned representation map, a causal role classifier, a claim-status filter, and an estimand lock. The lock fixes a standardized proximal effect before generated covariates enter the analysis. Under audit correctness and standard identification assumptions, admissible role assignments preserve this estimand. We apply the established conditional-covariance characterization of compression bias to substitution of generated representations for design-relevant states. A standardized decomposition separates compression, conditional-law, and standardization drift. Further results cover mediator adjustment, post-action leakage, marker-intervention conflation, outcome-guided discovery, and state-measurement error. Cluster-level orthogonal estimators distinguish empirical and superpopulation targets under repeated sessions and missing outcomes. Simulations show that refinement helps when it retains design-relevant information, whereas design erasure, leakage, and same-data marker selection can produce bias or undercoverage. The framework places causal semantics and claim status before confirmatory inference with generated representations.

1 Introduction

Generated covariates can alter a sequential experiment’s causal question when their roles are unspecified. The paper proposes causal typing and an estimand lock to govern their confirmatory use, with representation versioning to make that use auditable.

1 Introduction

  • A generated feature may be pre-action state or design information, a treatment version, or a post-action mediator, outcome proxy, observation process, or intercurrent event.Prediction alone does not establish causal admissibility.Source: §1 · pp. 1–2
  • Typing specifies the role required for a locked claim, while certification of that role in a particular data environment remains a separate problem.The paper studies roles one generated covariate at a time rather than jointly certifying a reconstructed pre-action information set.Source: §1 · p. 2
  • The framework defines versioned generated representations, assigns them causal roles, filters claim status, and locks a target contrast before confirmatory analysis.Admissible role assignments identify the locked standardized proximal effect under standard identification assumptions.Source: §1 · p. 2
  • The paper applies conditional-covariance identities to representation substitution and decomposes standardized drift into compression, conditional-law, and standardization components.The conditional form identifies design information a generated representation may discard.Source: §1 · p. 2
  • It derives a cluster-level orthogonal estimator for the locked effect with repeated sessions and missing outcomes, and organizes simulations around type violations.The simulation focus is type violations rather than disease-specific efficacy.Source: §1 · p. 2

2 Sequential experiment and generated representations

  • Representation maps may be manual, conventional, AI-based, or hybrid, and their versions record inputs, feature schema, preprocessing, quality control, and version date or hash.Mixed-window inputs are treated as post-action for primary total-effect analysis unless a separable pre-action component can be reconstructed and versioned.Source: §2 · p. 3

3 Causal types, claim status, and estimand locks

The causal type discipline assigns generated covariates roles and claim statuses, then uses an estimand lock to constrain which variables may enter a confirmatory analysis. Admissibility rules preserve the intended state-design object and distinguish violations that change the causal quantity from those that compromise the claim.

3 Causal types, claim status, and estimand locks

  • Causal types distinguish treatment, baseline state, candidate marker, history, design, mediator, outcome, observation, intercurrent-event, and excluded roles.A marker can support specified effect-modification analysis but does not by itself justify a claim about intervening on that marker.Source: §3 · p. 4
  • The estimand lock fixes treatment versions, outcome time, admissible pre-action state-design object, target distribution, intercurrent-event and observation strategies, and permitted claim status.Participant-weighted and session-weighted targets differ when participants contribute different numbers of sessions.Source: §3 · p. 5

4 Admissibility rules for generated covariates

  • When role evidence is weak, analysts should exclude the covariate from the confirmatory state-design object, assign it a conservative downstream role, or use an alternative lock.Outcome-guided covariates cannot support confirmatory effect-modification claims without an appropriate selection or inference procedure.Source: §4 · pp. 5–6
  • The confirmatory state-design object contains pre-action state and design variables, retains design information needed for assignment or observation, and excludes mediators from primary total-effect adjustment.Mechanism variables require a separate mediation lock.Source: §4 · p. 6
  • Mediator adjustment can shift a total-effect target, while reporting a state marker as an intervention conflates observed-state heterogeneity with effects of manipulating the state.The marker claim is a distinct problem from adjustment for a mediator.Source: §4 · p. 6
  • Design erasure can remove assignment, eligibility, availability, or observation information; post-action leakage and same-outcome selection can also undermine the locked analysis or its confirmatory claims.The listed consequences include possible failure of exchangeability or positivity after compression and loss of confirmatory support for nominal intervals and tests.Source: §4 · pp. 6–7

5 Identification and estimand preservation

Under audit correctness and standard identification conditions, admissible role assignments identify the locked standardized proximal effect. When outcome observation is not conditionally ignorable, the analysis instead requires sensitivity parameters.

5 Identification and estimand preservation

  • The preservation result assumes the audit correctly records source windows, representation versions, causal types, and claim status, with reproducible representations and required design information retained.The treatment versions must be operationally defined, and exchangeability and positivity must hold for the target region.Source: §5 · p. 7
  • If outcome observation is not conditionally ignorable, failure of observation exchangeability calls for a sensitivity family rather than a point claim.The stated sensitivity parameters describe observation probability and the mean difference between unobserved and observed potential outcomes by treatment arm and state.Source: §5 · p. 7
  • With admissible roles and Assumptions 1–6, the observed-data functional identifies the locked standardized proximal effect.The result also holds without the observation indicator when outcomes are complete.Source: §5 · p. 8

6 Estimand drift under type violations

Generated covariates can change a locked estimand when they replace design-relevant information or are assigned incompatible causal roles. The section characterizes compression drift and shows how several role violations alter the target contrast or claim.

6.1 No-free substitution and compression drift

  • Within a representation level, conditional-mean drift equals a covariance between outcome means and assignment probabilities; contrast drift is the difference between the arm-specific terms.A contrast is preserved when that difference is zero; zero drift in each arm is sufficient, but cancellation can also preserve the contrast.Source: §6.1 · p. 9

6.2 Design erasure as a special case

  • Predictive sufficiency alone cannot certify causal substitution, and replacing design information with a predictive representation preserves exchangeability only under a causal compression condition.Generated covariates should usually augment, rather than replace, variables relevant to assignment, availability, eligibility, observation, or positivity.Source: §6.1 · p. 9, §6.2 · p. 11

6.1 No-free substitution and compression drift

  • Standardized substitution drift separates conditional-compression, conditional-law, and standardization components, with the latter two able to change the estimand even when compression drift is zero.The conditional-law term vanishes when observed and target laws induce the same conditional distribution of W given Z; the standardization term vanishes when the analyst uses the locked push-forward.Source: §6.1 · p. 10

6.3 Mediator-as-state drift

  • Conditioning on a post-action mediator as if it were pre-action state generally changes the locked total proximal effect into a different estimand.In the stated linear model, the total effect is β_A + β_M α_A, while adjustment for M gives β_A; the discrepancy is −β_M α_A.Source: §6.3 · p. 11

6.4 Marker dependence is not marker intervention

  • The same observed distribution and state-dependent treatment effect can be compatible with different effects of intervening on the recorded marker.A cue appearing more effective at high marker values therefore supports a next-trial intervention hypothesis, not a conclusion that manipulating the marker improves outcomes.Source: §6.4 · pp. 11–12

6.5 Post-action leakage

  • Conditioning on a post-treatment representation compares treatment-specific strata, not the locked pre-action-state contrast or generally a common principal stratum.This applies to post-treatment conversations, post-action wearable segments, post-session notes, and outcome-adjacent text; narrative representations can leak semantically as well as temporally.Source: §6.5 · p. 12

6.6 Outcome-guided covariate selection and claim drift

  • Selecting the largest absolute statistic among p independent null tests and applying the naive 0.05 rule yields type I error above 0.05 for p > 1, approaching one as p grows.The selection calculation makes the claim-status filter consequential: outcome-guided discoveries remain exploratory absent pre-specification, sample splitting, simultaneous inference, or valid selective inference.Source: §6.6 · p. 13

6.7 Noisy state markers attenuate effect modification

  • Under classical state-measurement error, the estimated treatment-by-marker coefficient is attenuated by the reliability ratio Var(U)/(Var(U)+Var(e)).Differential error, state-adaptive treatment, or representation effects on selection and missingness can produce bias beyond simple attenuation.Source: §6.7 · p. 13

7 Orthogonal estimation of the locked effect

The section distinguishes empirical targets evaluated at observed states from superpopulation targets that also average over the sampling distribution of those states. It describes orthogonal, cluster-level estimation under identification and regularity conditions, while noting limits on conditional coverage and the need to justify allocation probabilities.

7 Orthogonal estimation of the locked effect

  • The theorem treats empirical and superpopulation targets separately, with the latter also averaging over the sampling distribution of target states.The empirical-target statement concerns marginal error about a random target and does not establish coverage conditional on the full sequence of states.Source: §7 · pp. 13–15
  • The type discipline determines which variables may enter nuisance functions; standard semiparametric machinery then estimates the locked effect.The section distinguishes empirical targets evaluated at observed target states from superpopulation targets.Source: §7 · pp. 13–14
  • The identification and nuisance-model requirements must use the same pre-action information, and observational allocation probabilities must be estimated or otherwise justified.A prespecified bounded session schedule may determine weights, but outcome-dependent realized session counts are not covered.Source: §7 · pp. 14–15
  • Normal confidence intervals require positive limiting variance, and the empirical-target result does not guarantee coverage conditional on the entire state sequence.Later states may contain earlier allocations and outcomes; conditional coverage requires an additional conditional central limit theorem.Source: §7 · p. 15
  • Cluster cross-fitting, orthogonal scores, product-rate conditions, and individual consistency support negligible estimation remainders and cluster-level asymptotic inference.The proof also invokes an independent-cluster central limit theorem and cluster-score variance consistency.Source: §7 · p. 15

8 Type uncertainty and audit discipline

Causal type assignment is a recorded, auditable judgment rather than an automatically validated output. When roles are ambiguous, the analysis should use a conservative downstream role or present separate locked estimands for competing roles.

8 Type uncertainty and audit discipline

  • The classifier records a causal judgment about a versioned covariate; audit exposes judgment but does not eliminate or automatically validate it.A wrong type assignment can still yield a wrong locked analysis.Source: §8 · pp. 15–16
  • When a covariate could have multiple roles, the primary analysis should use the most downstream role that can affect the estimand.Mixed pre-action and post-action summaries are treated as post-action unless their pre-action component is separable; outcome-adjacent notes default to type Y absent an independent source window.Source: §8 · p. 16
  • Outcome-trained representations are exploratory for effect modification unless training, selection, and inferential adjustment are external to or separated from confirmatory analysis.Variables governing availability, eligibility, randomization probability, or observation receive design or observation types even when outcome-predictive.Source: §8 · p. 16
  • If auditors disagree about a variable's role, the analysis should exclude it from the confirmatory state or report separate locked estimands for the competing roles.The paper treats such disagreement as disagreement about the causal question, not ordinary measurement error.Source: §8 · p. 16

9 Stress-test simulation

The stress-test simulations illustrate how generated-state choices and type violations can affect estimation of a standardized total effect. Retaining design information while refining state representation performs similarly to the oracle in the reported simple setting, whereas design erasure and post-action adjustment can distort results.

9 Stress-test simulation

  • The simulation is a reproducible artificial stress test of repeated-session type violations, not a proof of theorems or an assessment of clinical efficacy.Its setup includes adaptive assignment, design-relevant and generated states, post-action mediator and leakage variables, and optional observed-state missingness.Source: §9 · pp. 16–17
  • Retaining design information while adding the refined state performs similarly to the oracle benchmark, while mediator adjustment and post-action leakage produce large negative bias for the total effect.The simulations also report that state-adaptive assignment makes naive treatment contrasts unusable and that refinement alone is insufficient when it erases design-relevant information.Source: §9 · p. 17, Table 4 · p. 17
  • Design erasure is most damaging when omitted design information jointly predicts treatment assignment and potential-outcome means.The stress variants alter positivity, direct design effects on outcome means, and dependence of the refined state on design status; these stylized scenarios do not exhaust representation failure modes.Source: §9 · p. 17, Table 5 · p. 18
  • Under no true generated-covariate moderation, same-data selection of a candidate using its outcome-based interaction statistic led to severe undercoverage of nominal intervals.The study contrasts this with split-sample selection and estimation; the result illustrates why claim status matters.Source: §9 · pp. 17–18, Table 6 · p. 18

10 Relationship to existing literature

The paper builds on causal inference, generated-regressor, and trial-estimand literatures but focuses on the prior question of which causal role a generated covariate may play. Its contribution is integrating causal types, claim status, an estimand lock, and audit rules, rather than introducing the covariance mechanism or orthogonal score.

10 Relationship to existing literature

  • Where micro-randomized-trial literature defines and estimates proximal or excursion effects [21], this paper asks which generated covariates may define the state without changing the estimand.The framework draws on potential-outcome and longitudinal causal inference ideas.Source: §10 · p. 18
  • The paper's contribution is integrating causal types, claim status, an estimand lock, and audit rules, not discovering the covariance mechanism or zero-covariance compression criterion.The mathematical antecedents are deconfounding-score results by D’Amour and Franks [37] and Clivio et al. [38].Source: §10 · p. 18
  • The novelty is not the orthogonal score itself, but its integration with an estimand-preserving type discipline for generated covariates.The estimation section connects to semiparametric efficiency, double/debiased machine learning, and targeted learning.Source: §10 · pp. 18–19
  • Causal typing precedes generated-regressor inference: it assesses whether a variable belongs in the causal analysis, while downstream methods address uncertainty after role acceptance.The related methods address prediction error, first-stage uncertainty, measurement error, calibration, or local robustness [2] [1] [6]; Duan and Pelger describe moment-specific bias correction using a small human-labeled calibration set.Source: §10 · p. 19
  • Causal typing complements AI trial reporting and regulatory guidance by recording the inferential role a generated variable may play in the locked analysis.The related guidance emphasizes transparent AI reporting and context of use, credibility assessment, governance, documentation, and life-cycle management.Source: §10 · p. 19

11 Implementation as an audit object

The audit record makes causal-role decisions reproducible by recording the estimand lock, timing, representation versions, claim status, and admissibility rationale. Its simulation-grounded examples distinguish admissible refinement from design erasure and position the audit before downstream representation analysis.

11 Implementation as an audit object

  • The audit record captures the estimand lock, action milestone, source windows, representation versions, causal types, claim status, and admissibility decisions with reasons.It can support analysis plans, data dictionaries, codebooks, pipelines, and software without prescribing a specific implementation.Source: §11 · p. 19
  • Table 7 is a simulation-grounded audit of stress-test variables, not a clinical vignette or patient record.Its entries are defined by the data-generating process.Source: §11 · pp. 19–20, Table 7 · p. 20
  • The admissible-refinement specification adds U(1) only after retaining D, whereas design erasure substitutes for design-relevant information.These examples mark how the same refinement can be admissible or problematic depending on its relation to design information.Source: §11 · p. 20
  • The audit determines whether a representation may play its proposed causal role, unlike generated-regressor methods or text-as-data methods, which address downstream analysis or representation measurement.It records whether a representation is state, design information, mediator, outcome proxy, observation process, exploratory signal, or excluded material under the lock.Source: §11 · p. 20

12 Reverse audit for hypothesis generation

Reverse audit preserves type-violation signals for exploratory or future work without changing the current estimand or certifying a new causal effect. The broader discussion and conclusion position causal typing as a gate for confirmatory claims, while identifying limits, target choices, and reproducibility scope.

12 Reverse audit for hypothesis generation

  • Reverse audit preserves inadmissible covariates as hypothesis-queue signals without changing the current estimand or certifying a new causal effect.Post-action features may motivate separate mediation or mechanistic work, while leakage and design-replacement signals require temporal validation and are not proof of admissibility.Source: §12 · pp. 20–21
  • Type violations can reflect hidden mechanisms or errors in timing, documentation, coding, observation, or feature extraction, so reverse-audit outputs remain exploratory or next-trial unless a new lock and valid design or adjustment are supplied.Turning a downgraded signal into a randomized treatment version requires operational versions, feasibility constraints, and a separate design calculation.Source: §12 · p. 21

13 Discussion

  • Generated covariates and clinical narratives require causal-role specification before confirmatory claims; prediction and clinically plausible interpretation alone do not establish an intervention effect.Roles such as pre-action state, mediator, treatment version, outcome proxy, or design variable can change the causal question if left unspecified.Source: §13 · p. 22
  • Fine-grained generated representations can remain admissible state markers or effect modifiers when pre-action, version-stable, and compatible with the locked state-design object.The framework favors augmenting rather than replacing design-relevant variables, and routes post-action features to their appropriate roles instead of pre-action adjustment.Source: §13 · p. 22
  • Causal typing does not discover causal structure, license interventions on state markers, solve unmeasured confounding, or make high-dimensional conditioning easy.It routes candidates to confirmatory, secondary, exploratory, mediation, design-audit, or future-intervention uses; representation noise and model-version drift remain separate concerns.Source: §13 · pp. 22–23
  • Causal typing depends on human judgment, and the local role theory does not certify jointly reconstructed pre-action information sets in retrospective environments.The discussion also flags uncertain action milestones, sufficiency assumptions for exchangeability and observation, unmeasured states, interference, missing-not-at-random mechanisms, and stylized simulations.Source: §13 · p. 23

14 Conclusion

  • The conclusion extends the framework beyond AI-derived clinical covariates to longitudinal data-derived representations and encoded human clinical narratives.Non-confirmatory signals can be retained as mechanism hypotheses, early-response signals, unrecorded-design signals, or next-trial candidates, not as shortcuts to causal discovery.Source: §14 · pp. 23–24

Data and code availability

  • The paper is a theoretical proposal with artificial simulations and uses no real patient data; its reproducibility package includes scripts, replicate results, verification materials, and hashes.The framework is software-agnostic.Source: Data and code availability · p. 24

B Proof details for orthogonal estimation

The proof establishes orthogonal estimation through weighted cluster residual identities and conditional fold-level error control. It also shows why weights must obey predictability restrictions: trajectory-dependent weighting can distort even a zero-effect contrast.

B Proof details for orthogonal estimation

  • Fixed predictable weights measurable from the pre-action information support mean-zero weighted cluster residuals and the nuisance-derivative cancellation.An unweighted mean-zero calculation alone does not justify the step for arbitrary trajectory-dependent weights.Source: B Proof details for orthogonal… · p. 25
  • Conditional training-fold arguments and cluster-score convergence yield the theorem’s asymptotic expansions, normal limits, and variance consistency.The proof uses independent clusters and L2 convergence of estimated cluster contributions.Source: B Proof details for orthogonal… · p. 25
  • With two sessions and zero treatment effects, using later history to define a weight gives the true first-session contrast score a weighted expectation of 1/2.The example uses complete observation and an independent symmetric-sign outcome component.Source: B Proof details for orthogonal… · p. 25

C Simulation reproducibility

The appendix documents simulation regeneration and uncertainty, then provides an audit checklist and structured template for specifying timing, targets, roles, and validation. These materials support reproducibility and explicit causal-role decisions without prescribing a file format.

C Simulation reproducibility

  • The simulation tables were regenerated from the reproducibility package using 1,000 Monte Carlo replications, with replicate-level CSV summaries.At 95% coverage, the approximate Monte Carlo standard error is 0.0069, so coverage differences around one percentage point warrant caution.Source: C Simulation reproducibility · p. 26
  • The simulation’s observation mechanism depends on observed pre-action and history variables; a latent-state mechanism is only a sensitivity stress test, not observed-data MAR.The package includes scripts, replicate results, summary tables, logs, verification, and file hashes.Source: C Simulation reproducibility · p. 26

D Checklist for a causal type audit

  • The checklist asks analysts to specify the action time, source-window overlap, target distribution, covariate roles, and which features were selected after viewing outcomes.It also asks which variables determine assignment, availability, eligibility, observation, or positivity, and which post-action variables are excluded from primary total-effect adjustment.Source: D Checklist for a causal type audit · p. 26
  • The checklist covers claim status, representation versioning, safe-compression arguments, uncertain type assignments, measurement or extraction error, and follow-up validation for downgraded signals.Follow-up may require a new lock, temporal or external validation, sample splitting, selective inference, or a randomized intervention version.Source: D Checklist for a causal type audit · pp. 26–27

E Structured audit-template fields

  • The structured template records treatment and timing details, target distribution, intercurrent-event and missingness strategies, feature source and version, causal type, admissibility, violation, and validation needs.It records the essential fields without prescribing a file format.Source: E Structured audit-template fields · p. 27
Loading 2609.17772v1…