PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual-Based Causal Discovery via Pretrained In-Context Quantile Regression

Daniel Klippert, Quang Nhat Huynh, Alexander Marx

arXiv:2610.29530v1stat.MLcs.LG

TL;DR

Residual-based causal discovery for location-scale noise models has lacked a nonparametric residual-estimation approach, while RESIT was developed for additive-noise models. LOSCAR extends RESIT using quantile regression with pretrained tabular foundation models, and its quantile-based variants outperform FGLS across experiments while reaching state-of-the-art performance on established bivariate benchmarks.

  • Problem

    Existing LSNM methods rely on distributional or noise-symmetry assumptions, while RESIT's nonparametric residual approach was developed for ANMs.

    Source: §1 · p. 1

  • Method

    LOSCAR extends RESIT to LSNMs by estimating conditional mean and scale through quantile regression, using pretrained tabular foundation models as plug-in estimators.

    Source: §1 · p. 1, §2.3 · p. 3

  • Results

    Across experiments, quantile-based variants consistently outperform FGLS, achieve state-of-the-art performance on established bivariate benchmarks, and remain on par with state-of-the-art baselines in low- to medium-dimensional settings.

    Source: §6 · p. 8

  • Takeaways & Limitations

    In-context quantile regression provides a nonparametric way to estimate conditional mean and scale for residual-based causal discovery using pretrained tabular foundation models.

    Source: §1 · p. 1

    LOSCAR's scalability is limited by repeated model-fitting costs and conditional-independence testing with large conditioning sets, and its greedy search may propagate errors across steps.

    Source: §6 · p. 8

Abstract

from arXiv · show

Residual-based causal discovery assumes that each variable is generated from its parents and an independent noise source. It estimates this noise and tests its independence from candidate causes. For identifiable functions such as additive-noise models (ANMs) and the broader class of location-scale noise models (LSNMs), this principle identifies the causal direction, as only the causal model admits this independence. Existing LSNM methods rely on parametric assumptions to obtain standardized residuals, risking model misspecification. In contrast, the RESIT-framework allows for nonparametric estimation of residuals to iteratively identify sink nodes, but was proposed for ANMs. To close this gap, we propose Loscar, an extension of RESIT to LSNMs that estimates the conditional mean and scale via quantile regression, enabling pretrained tabular foundation models as plug-in estimators. We compare it with a feasible generalized least squares (FGLS) variant, which is consistent given suitable basis functions. Across bivariate and multivariate causal-discovery benchmarks, in-context quantile regression substantially outperforms FGLS with flexible function classes, particularly for multivariate causes.

1. Introduction

The paper extends residual-based causal discovery from additive-noise models to location-scale noise models using pretrained tabular foundation models as plug-in estimators. Across bivariate and multivariate benchmarks, its quantile-regression variants outperform FGLS in flexible settings and perform competitively against state-of-the-art methods.

1. Introduction

  • Unlike amortized causal-discovery approaches that require extensive pretraining and lack formal guarantees, LOSCAR uses a pretrained tabular foundation model as an estimator.The introduction situates this approach alongside causal-discovery methods that train on synthetically generated data.Source: §1 · p. 1
  • LOSCAR extends RESIT to LSNMs by estimating conditional mean and standard deviation through quantile regression, enabling pretrained tabular foundation models as plug-in estimators without retraining or fine-tuning.The paper also compares two quantile-regression strategies with an FGLS variant that consistently estimates conditional mean and standard deviation given appropriate basis functions.Source: §1 · pp. 1–2
  • Across 19 bivariate benchmarks, LOSCAR achieves 90.04% accuracy and outperforms the spline-based FGLS variant, which struggles with complex location-scale relationships.LOSCAR-GL performs on par with LOSCAR.Source: §1 · p. 2, Figure 1 · p. 2

2. Residual-Based Causal Discovery

Residual-based discovery estimates noise and tests whether it is independent of candidate causes; for identifiable LSNMs, this supports causal-direction recovery. LOSCAR adapts RESIT to LSNMs by estimating standardized residuals and iteratively identifying sink nodes.

2.1. Location-Scale Noise Models

  • An LSNM models a variable with parents as a deterministic location function plus a positive scale function multiplying noise; ANMs are the special case with constant scale one.The LSNM identifiability results cited in the paper extend to multivariate settings, subject to pathological cases.Source: §2.1 · p. 2

2.2. Cause-Effect Inference by Residual-Based Independence Testing

  • For an identifiable LSNM, the causal direction admits a residual independent of its cause, whereas the reverse direction does not admit such an independence relation.RESIT uses residual-based independence tests to choose between candidate directions; if both directions reject independence and a decision is forced, it selects the direction with the higher p-value.Source: §2.2 · pp. 2–3
  • Consistent estimates of conditional mean and standard deviation suffice to recover standardized noise independent of the cause when the noise has finite first two moments.The estimates need not equal the structural location and scale functions when noise is not centered or unit-scaled; standardization cancels those differences.Source: §2.2 · p. 3

2.3. Multivariate Causal Discovery

  • LOSCAR adapts RESIT to LSNMs, using estimated standardized residuals to infer multivariate causal graphs.Its two phases learn a causal ordering and then prune superfluous edges.Source: §2.3 · p. 3
  • In the ordering phase, LOSCAR iteratively selects the remaining node whose standardized residuals have the largest independence-test p-value against its candidate parents.Conditional mean and standard deviation are estimated using the other remaining nodes as predictors.Source: §2.3 · p. 3
  • LOSCAR assumes an identifiable LSNM and uses HSIC to test residual independence; consistency also requires suitable noise estimation and independence testing.The stated consistency conditions include causal minimality and an independence oracle for original RESIT, with sample-splitting or strong consistency conditions noted for HSIC.Source: §2.3 · pp. 3–4

3. Estimating Methods

LOSCAR estimates conditional mean and scale using in-context quantile regression or spline-based FGLS to extract residuals under LSNMs. The quantile approaches use pretrained TabPFN without further fine-tuning, while FGLS is consistent under suitable modeling assumptions.

3. Estimating Methods

  • LOSCAR compares two quantile-based scale estimators using TabPFN with a spline-based feasible generalized least squares procedure that is consistent under suitable modeling assumptions.The quantile approaches recover scale from the estimated conditional quantile function; FGLS jointly estimates mean and variance.Source: §3 · p. 4

3.1. Pretrained In-Context Quantile Regression

  • Under symmetric noise, the conditional mean and median coincide, allowing residual extraction from the quantile function while estimating scale.Source: §3.1 · p. 4

3.1.1. QUANTILE-DIFFERENCE SCALE ESTIMATION

  • Two conditional quantiles recover the scale exactly when the noise quantiles are known, and recover it up to scale otherwise, which suffices for LOSCAR.The implementation assumes standard normal noise quantiles only to fix an otherwise irrelevant multiplicative constant.Source: §3.1.1 · p. 4

3.1.2. GAUSS-LEGENDRE SCALE ESTIMATION

  • Gauss-Legendre quadrature estimates conditional variance from the quantile function and conditional mean, with the integral approximation exact for correctly estimated polynomial quantiles of degree at most 2m −1.The default uses m = 10 nodes; estimating mean errors directly propagate to the variance estimate.Source: §3.1.2 · p. 5

3.1.3. SUITABILITY OF TABPFN

  • TabPFN-3 supplies conditional quantile estimates in context without further fine-tuning, and its predicted CDF is inverted to decode quantiles at inference.The estimates are for training points in the provided context; sample splitting is suggested as a possible future consideration.Source: §3.1.3 · p. 5

3.2. Consistent Estimation via Feasible GLS

  • The FGLS model uses fixed-dimension spline bases for the conditional mean and variance, with the variance basis excluding an intercept; its estimated variance and residuals are consistent up to scale.A small ridge penalty is used in each regression step for numerical stability.Source: §3.2 · pp. 5–6

4. Related Work

Prior work establishes identifiability for several restricted causal model classes, including LSNMs, and includes quantile-based scoring approaches. LOSCAR differs from amortized methods by leveraging pretrained tabular models without extensive training and exploiting LSNM identification results.

DAG Identifiability

  • Identifiability results cover linear non-Gaussian models, causal additive models, ANMs, PNL models, and LSNMs; PNL models and LSNMs include ANMs as a special case.Source: §4 · p. 6

Quantile-Based Causal Discovery

  • Quantile-based causal discovery includes QCCD's MDL score and a quantile-partial-effects score for learning causal order, whose estimation depends on sensitive hyperparameters.Source: §4 · p. 6

Causal Discovery with LSNMs

  • Existing LSNM methods use symmetric-noise skewness, Gaussianity, or parametric heavy-tailed or skew-normal noise assumptions; ROCHE is limited to bivariate use.Source: §4 · p. 6

Amortized Causal Discovery

  • Unlike amortized causal discovery trained on simulated causal networks, LOSCAR uses pretrained quantile-capable tabular models without extensive training and directly exploits LSNM identification results.Source: §4 · p. 6

5. Experiments

The experiments compare LOSCAR variants with causal-discovery baselines on bivariate and multivariate benchmarks. Quantile-based variants lead the bivariate accuracy results and are competitive in multivariate LSNM experiments, with outcomes varying by setting.

5. Experiments

  • The bivariate evaluation covers 19 established benchmarks, while multivariate experiments include synthetic and real-world datasets.Bivariate metrics are accuracy and AUDRC; multivariate metrics are F1, order errors, SHD, and SID.Source: §5 · p. 6, §5.1 · p. 7, §5.2 · p. 7

5.1. Bivariate Experiments

  • 90.04% accuracy for LOSCAR is the best bivariate result, versus 89.81% for LOSCAR-GL and 74.97% for LOSCAR-FGLS.ROCHE is the most competitive baseline on accuracy at 89.52%, while it exceeds LOSCAR by 0.44% on AUDRC.Source: §5.1 · p. 7, Table 1 · p. 7

5.2. Multivariate Experiments

  • Multivariate synthetic datasets use Erdős–Rényi graphs with d ∈ {6, 12}, b = d edges, and 1,000 samples per dataset.For each graph size, the evaluation generates 100 ANM and 100 LSNM datasets.Source: §5.2 · p. 7
  • On multivariate LSNM data, FOUNDCAUSE performs best overall, while LOSCAR and LOSCAR-GL remain competitive and LOSCAR-FGLS is not competitive.For d = 6, LOSCAR variants lead classical baselines on F1, order errors, and SID, while SKEWD leads on SHD; for d = 12, SKEWD is generally comparable or slightly better, particularly in F1.Source: §5.2 · pp. 7–8, Figure 2 · p. 8
  • On Light Tunnel, FOUNDCAUSE and CDFM achieve 0 and 1 order errors, respectively, compared with 12 for LOSCAR, the second-best result.LOSCAR and LOSCAR-GL are the strongest classical baselines across metrics; SKEWD did not terminate after more than 72 hours.Source: Table 2 · p. 7, §5.2 · p. 8

6. Conclusion

LOSCAR uses nonparametric in-context quantile regression to estimate LSNM residuals. The quantile-based variants consistently outperform FGLS in the experiments, while LOSCAR reaches state-of-the-art bivariate performance and remains on par with baselines in low- to medium-dimensional settings.

6. Conclusion

  • Across experiments, quantile-based LOSCAR variants consistently outperform FGLS, achieve state-of-the-art bivariate results, and remain on par with baselines in low- to medium-dimensional settings.Scalability is limited by repeated model fitting and conditional-independence testing with large conditioning sets; greedy search may propagate errors across steps.Source: §6 · p. 8

A.1. Metrics

  • AUDRC measures how well confidence aligns with correct outputs, is bounded by 0 and 1, and has better performance at higher values.Benchmark averages are weighted equally, regardless of the number of pairs in each benchmark.Source: §A.1 · p. 12
  • Order errors count edges that cannot be recovered because of the chosen causal ordering, with zero errors for a correct ordering.The ordering is obtained by topologically sorting each method’s predicted graph.Source: §A.1 · p. 12
  • SHD counts predicted edges that are missing or absent in the target graph, counting a reversed edge as two errors.F1, precision, and recall compare predicted and ground-truth directed edges; SID measures graph closeness in causal-inference statements.Source: §A.1 · p. 12

A.2.1. GENERATED SYNTHETIC DATASETS

The synthetic multivariate datasets vary graph size, structural functions, and noise distributions across ANM and LSNM settings.

A.2.1. GENERATED SYNTHETIC DATASETS

  • For d ∈ {6, 12}, the synthetic graphs have b = d edges, with 100 ANM and 100 LSNM datasets of 1,000 samples generated per graph size.The structural functions are sums of random sigmoidal functions over each variable’s parents.Source: §A.2.1 · pp. 12–13
  • Node noise is sampled from Gaussian, exponential, or skew-t distributions, with each node’s distribution chosen uniformly in the main experiments.The noise variance lies in (1, 2); for non-source nodes it is scaled by 0.2 in ANM settings and 0.35 in LSNM settings.Source: §A.2.1 · p. 13

A.2.2. BIVARIATE BENCHMARK DATASETS

The appendices describe the bivariate benchmarks, real-world datasets, implementation choices, and baseline configurations used in the evaluation.

A.2.2. BIVARIATE BENCHMARK DATASETS

  • The bivariate evaluation draws on SIM and Tübingen pairs, cause-effect datasets from Goudet et al., synthetic ANM, LSNM, and multiplicative-noise datasets, and skewed-noise extensions.The Tübingen pairs exclude binary variables and use recommended weighting; cause and effect are standardized by default.Source: §A.2.2 · p. 13

A.3. Real-World Multivariate

  • The real-world evaluation uses observational Sachs measurements and 10,000 observational samples from the Light Tunnel system.Sachs contains 11 variables and 853 data points; its ground-truth network has been challenged. Light Tunnel has 20 nodes and 39 edges.Source: §A.3 · p. 14

A.4. Hyperparameter Configuration LOSCAR

  • LOSCAR uses TabPFN-3 for conditional means and quantiles, estimating scale from quantile differences by default or via Gauss–Legendre quadrature in LOSCAR-GL.The default quantile levels are 0.05 and 0.95; LOSCAR-GL uses 10 quantiles.Source: §A.4 · p. 14
  • LOSCAR-FGLS and RESIT-OLS estimate means with ridge regression and cubic-spline bases, while LOSCAR-FGLS also estimates variance using a spline basis.The configured mean basis has 8 features per covariate and the variance basis has 6.Source: §A.4 · p. 14

A.5. Baselines

  • The baseline suite uses published defaults for most methods, with pruning enabled for CAM and the likelihood-based SKEWD variant used for comparison.The multivariate SKEWD optimization step was omitted for scalability; FOUNDCAUSE and CDFM use prediction-probability differences to force bivariate decisions.Source: §A.5 · pp. 14–15

B.1.1. DETAILED BIVARIATE EXPERIMENTAL RESULTS

Across bivariate benchmarks, quantile-based LOSCAR variants perform well across datasets, while LOSCAR-FGLS drops substantially on LSNM data; skewness affects several methods differently.

B.1.1. DETAILED BIVARIATE EXPERIMENTAL RESULTS

  • 37.5%, 58.0%, and 72.0%: LOSCAR-FGLS accuracy rises as skewness increases across LSs-med, LSs-str, and LSs-ext.This contrasts with CAM, QCCD, and CDFM, whose performance degrades as skewness increases, while other methods remain stable.Source: §B.1.1 · p. 15
  • Quantile-based LOSCAR variants perform well across the benchmark datasets, whereas LOSCAR-FGLS performs well for additive noise but drops substantially on LSNM data.The authors attribute the LSNM drop to inappropriate variance estimates.Source: §B.1.1 · p. 15
  • FOUNDCAUSE reaches almost perfect accuracy on multiplicative uniform noise, while CDFM reaches 68% on SIMc and FOUNDCAUSE reaches 47%.The authors suggest these differences may reflect differences in the training datasets used for the causal foundation models.Source: §B.1.1 · p. 15

B.1.2. ABLATION STUDIES BIVARIATE BENCHMARK

The bivariate ablations find LOSCAR relatively robust to parameter choices, while LOSCAR-GL has some benchmark-specific drops. TabPFN-3 outperforms version 3.5 in this framework, with smoother mean and quantile estimates.

B.1.2. ABLATION STUDIES BIVARIATE BENCHMARK

  • Across all bivariate benchmarks, LOSCAR remains within a 2.5% difference of the main variant; LOSCAR-GL shows similar results, with approximately 5% performance drops on Tübingen and LSs-ext.Source: §B.1.2 · p. 16
  • Increasing m from 10 to 100 does not improve performance overall, as noisier estimated quantiles near levels 0 and 1 can impair the integral approximation.The reported sensitivity to m is small; the ablation compares mean and standard-deviation estimates for one LSs benchmark pair.Source: §B.1.2 · p. 16
  • LOSCAR-FGLS is clearly outperformed by TabPFN-supported variants on flexible location-scale data, although FGLS improves under high skewness.The ANM baselines are comparable to one another but are outperformed by LOSCAR's location-scale variants.Source: §B.1.2 · p. 16

B.1.3. REMARKS TO TABPFN VERSION 3.5

  • TabPFN-3 performed better than TabPFN-3.5 in this framework, and the authors traced the difference to smoother conditional-mean and quantile estimates from version 3.The comparison used estimates on Pair 1 of the LSs dataset.Source: §B.1.3 · p. 17

B.2.1. RESULTS ON THE SACHS DATASET

The appendix reports distinct outcomes across real-world, synthetic LSNM and ANM, and runtime comparisons. Sachs results are mixed across metrics; LOSCAR variants are competitive on LSNMs, while LOSCAR-FGLS leads its quantile counterparts on mixed ANMs.

B.2.1. RESULTS ON THE SACHS DATASET

  • On Sachs, LOSCAR and LOSCAR-GL match CAM in order errors but have SHD values 2 and 4 higher than HOST, respectively; amortized methods perform comparably to LOSCAR variants.CAM and HOST perform best despite weak performance on Light Tunnel.Source: §B.2.1 · p. 17

B.2.2. TABULAR COMPARISON UNDER MIXED LOCATION-SCALE NOISE

  • On the Sachs LSs dataset, TabPFN-3 estimates are smoother than TabPFN-3.5 estimates; on heterogeneous synthetic LSNMs, LOSCAR variants generally obtain higher precision than recall.The tabular comparison reports mean F1, precision, recall, order errors, SHD, and SID.Source: §B.2.2 · p. 17, Figure 4 · p. 18

B.2.3. COMPARISON UNDER MIXED ADDITIVE NOISE

  • On heterogeneous synthetic ANMs, CAM is the strongest conventional baseline; at d = 6 its F1 is 0.839, versus 0.612, 0.632, and 0.718 for LOSCAR, LOSCAR-GL, and LOSCAR-FGLS.The best order-error and SID results come from FOUNDCAUSE or CDFM, while CAM and SKEWD outperform amortized baselines on F1 and SHD.Source: §B.2.3 · p. 18, Table 9 · p. 20
  • Unlike on mixed LSNMs, LOSCAR-FGLS outperforms the quantile variants on mixed ANMs, although all LOSCAR variants drop notably as dimensionality rises from d = 6 to d = 12.CAM is barely affected in F1, while FOUNDCAUSE improves in recall and order errors as dimensionality increases.Source: §B.2.3 · p. 18

B.2.4. RUNTIME COMPARISON UNDER MIXED NOISE DISTRIBUTION

  • For d = 20 Light Tunnel, SKEWD did not finish within 72 hours, whereas LOSCAR and LOSCAR-GL took around 85 and 110 minutes, respectively.For d = 12 ANMs, LOSCAR took about 2 minutes and LOSCAR-GL at most about 75 seconds; amortized methods finished below one second.Source: §B.2.4 · p. 20, Table 10 · p. 20

B.2.5. COMPARISON UNDER HOMOGENEOUS NOISE DISTRIBUTIONS

Under homogeneous noise distributions, the relative performance of LOSCAR variants depends on whether the data follow LSNMs or ANMs and on the noise distribution. Quantile-based variants are stable and strong across asymmetric LSNM noise, but Gaussian LSNMs favor HOST and FOUNDCAUSE.

B.2.5. COMPARISON UNDER HOMOGENEOUS NOISE DISTRIBUTIONS

  • On LSNMs, LOSCAR and LOSCAR-GL are strongest conventional baselines under exponential noise, while FOUNDCAUSE slightly outperforms them; under skew-t, they lead at d = 6 but trail SKEWD and FOUNDCAUSE slightly at d = 12.Under Gaussian LSNMs, HOST beats all LOSCAR variants across metrics and dimensions, while FOUNDCAUSE performs best overall.Source: §B.2.5 · p. 21, Figure 7 · p. 22, Figure 8 · p. 23
  • Quantile-based LOSCAR variants show stable, strong performance across the different asymmetric LSNM noise distributions, whereas LOSCAR-FGLS performs poorly in the LSNM setting.The authors relate this pattern to the nonparametric nature of the quantile-based method.Source: §B.2.5 · p. 21
  • On ANMs, LOSCAR-FGLS beats the quantile variants under exponential and Gaussian noise, but this ordering reverses under skew-t; LOSCAR variants are comparatively worse under Gaussian and skew-t noise.For exponential ANMs, several baselines lead; under Gaussian and skew-t, CAM and other baselines are strongest.Source: §B.2.5 · p. 21, Figure 10 · p. 25, Figure 11 · p. 26
Loading 2610.29530v1…