PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Space Selection and Heterogeneous Effect Estimation for Blood-Brain Barrier Permeability: A Random Forest to the Generalized Random Forest Pipeline

Tshemollo Rapolai, Seite Makgai, Mohammad Arashi

arXiv:2609.29076v1stat.MLcs.LG

TL;DR

BBB permeability prediction is important for CNS drug discovery, while evidence on whether lipophilicity’s association with permeability varies across chemical space is limited. The study compares feature representations and algorithms, then uses an orthogonalized forest to estimate heterogeneous associations. After accounting for observed confounding, it finds no evidence of systematic variation in that association across compounds in this dataset.

  • Problem

    It remains unclear whether lipophilicity’s association with BBB permeability varies across chemical space, and observational estimates are complicated by correlated molecular descriptors.

    Source: §1 · p. 2

  • Method

    The study compares molecular feature representations and algorithms, then estimates heterogeneous lipophilicity associations using an orthogonalized Generalized Random Forest on the selected feature space.

    Source: §1 · p. 3

  • Results

    After orthogonalization, no evidence remained that the log P–BBB permeability association varies systematically across compounds in this dataset.

    Source: §6.4 · p. 16

  • Takeaways & Limitations

    Within this dataset, the apparent heterogeneity from the naive estimator is attributed to confounding by molecular descriptors correlated with log P.

    Source: §6.4 · p. 16

    The preliminary naive estimates are only a baseline because log P is correlated with covariates that also influence permeability, including TPSA and heteroatom count.

    Source: §6.2 · p. 13

Abstract

from arXiv · show

Predicting blood-brain barrier (BBB) permeability is critical for central nervous system drug discovery. Using the MoleculeNet BBBP dataset (n = 2039), this study systematically ablates molecular feature spaces to isolate featurisation from model architecture. We evaluate three feature families (Morgan fingerprints, RDKit physicochemical descriptors, SMILES bigrams) across four learning algorithms. Results demonstrate that predictive performance depends jointly on feature representation and algorithm. Dynamic Random Forest using combined features achieved the highest mean AUC (0.970, 95% CI: 0.963-0.977). Second, this optimal representation enables exploratory estimation of heterogeneous associations between molecular structure and BBB permeability using Generalized Random Forests. Constructing a pseudo-treatment from a LogP median split, we applied double/debiased machine learning to account for confounding. Orthogonalization substantially attenuates the heterogeneity detected by naive causal forests; no conditional effects remained significant after false discovery rate correction (smallest adjusted p = 0.082). Furthermore, orthogonalized feature importance shifted toward residual structural information in SMILES bigrams. Ultimately, once observed confounding is properly accounted for, evidence that LogP-BBB associations vary systematically across chemical space is insufficient. This underscores that feature representation and model architecture are coupled design choices, and that unorthogonalized causal forests risk overstating genuine treatment effect heterogeneity.

1. Introduction

BBB permeability prediction matters for CNS drug discovery, but prior work often leaves feature representation entangled with algorithm choice. This study separates those choices and explores whether lipophilicity’s association with permeability varies across chemical space.

1. Introduction

  • BBB permeability is important in CNS drug discovery because the barrier restricts many therapeutic compounds from reaching brain tissue.Accurate prediction can support lead prioritization and reduce late-stage attrition.Source: §1 · p. 2
  • The paper evaluates AUC-ROC as its primary metric because the binary classification data are typically imbalanced and AUC is threshold-independent.AUC is described as less sensitive to imbalance than overall accuracy.Source: §1 · p. 2
  • Prior BBB prediction studies commonly use Random Forests and SVMs, while the contribution of feature representation is rarely separated from algorithm choice.Tree ensembles have often outperformed linear classifiers on this task.Source: §1 · p. 2
  • Most prior computational studies treat each compound as a fixed classification point, leaving open whether lipophilicity’s relationship with permeability varies across scaffolds and molecular properties.An average association may mask such variation.Source: §1 · p. 2
  • Because molecular descriptors correlated with lipophilicity also influence permeability, observational heterogeneity estimates risk conflating genuine variation with residual confounding.The study therefore uses an orthogonalized causal forest that residualizes both outcome and treatment on covariates before estimating effects.Source: §1 · pp. 2–3
  • The study jointly aims to identify an efficient predictive feature representation and explore heterogeneous associations between molecular lipophilicity and BBB permeability.The selected representation supplies the covariate space for the subsequent heterogeneity analysis.Source: §1 · p. 3

2. Ablation Study

The ablation compares molecular feature representations and learning algorithms for BBB permeability prediction, then selects a parsimonious representation for heterogeneity analysis. Bigrams contribute strongly to predictive performance, and their advantage over combinations without bigrams persists under scaffold-based evaluation.

2.1. Task and Dataset

  • The dataset contains 2,039 SMILES-labeled molecules, with 76% positive and 24% negative cases; the study reports AUC, F1, and balanced accuracy rather than raw accuracy alone.SMILES are the sole input; no 3D coordinates, molecular graphs, or precomputed descriptors enter the feature-engineering pipeline.Source: §2.1 · p. 3

2.2. Molecular Feature Spaces

  • Three SMILES-derived feature families—Morgan fingerprints, RDKit physicochemical descriptors, and TF-IDF SMILES bigrams—form seven singleton, pairwise, and full-combination spaces.The spaces enable evaluation of standalone signal, pairwise synergy, and redundancy.Source: §2.2 · p. 4

2.3. Classifiers

  • Four classifiers span linear and nonlinear approaches: Logistic Regression, Random Forest, RBF-kernel SVM, and Dynamic Random Forest.Dynamic RF adds trees incrementally and stops when out-of-bag performance fails to improve beyond a tolerance for consecutive rounds.Source: §2.3 · pp. 4–5

2.4. Bootstrap Evaluation Protocol

  • The evaluation varies feature space, sample size, and classifier independently, using 1,000 stratified bootstrap replicates per sample size and a fixed held-out test set.It summarizes mean AUC, variance, confidence intervals, and relative efficiency.Source: §2.4 · p. 5

2.5. Results

  • RF AUC is 0.972 for M+D+B, while D+B and M+B each reach 0.968 and M+D reaches 0.947; tree ensembles outperform Logistic Regression and SVM across representations.The paired M+D+B versus D+B comparison is not significant for RF (mean difference 0.003, 95% CI: −0.002–0.009, p = 0.204) or Dynamic RF (0.003, 95% CI: −0.004–0.010, p = 0.386).Source: §2.5 · pp. 6–7, §2.6 · p. 7, Table 1 · p. 7

2.6. Feature Space Selection for HTE Analysis

  • D+B with RF was selected for heterogeneity analysis because its performance was statistically equivalent to M+D+B while using 269 rather than over 2,000 dimensions.The lower-dimensional space reduces covariate dimensionality and supports interpretation without measurable predictive cost.Source: §2.6 · p. 7

2.7. Robustness to Scaffold-Based Splitting

  • Across 200 scaffold splits, M+D+B with RF has mean AUC 0.9591 (CI 0.9470–0.9735), down from 0.972 under random splitting; the ordering of feature spaces is preserved.Scaffold-split confidence intervals are wider, with width 0.0265 for M+D+B/RF versus 0.011 under random splitting.Source: §2.7 · pp. 8–9, Table 2 · p. 8
  • Under scaffold splitting, M+D+B with RF reaches mean AUC 0.9591, while D+B (0.9551) and M+B (0.9542) remain statistically indistinguishable from it and M+D (0.9247) lags.The central random-split pattern is preserved under this more demanding generalization test.Source: §2.7 · pp. 8–9, Table 2 · p. 8

3. Generalized Random Forests

Generalized Random Forests estimate parameters from local moment conditions, with splits designed to expose variation across covariate profiles. For treatment-effect estimation, they allow conditional effects to vary, while relying on assumptions that observational data cannot guarantee.

3. Generalized Random Forests

  • GRFs solve local moment conditions for the parameter of interest and split to maximize heterogeneity in that parameter.Unlike conventional Random Forests, which minimize squared prediction error, GRF splits seek subpopulations with materially different parameter estimates.Source: §3 · pp. 9–10

3.1. Treatment Effect Estimation

  • For treatment-effect estimation, the target is the Conditional Average Treatment Effect (CATE), which may vary across the covariate space.This contrasts with assuming a constant treatment effect.Source: §3.1 · p. 10
  • Honest estimation separates each tree’s bootstrap sample into structure and estimation subsets, so observations do not serve both roles.This improves statistical validity and asymptotic properties.Source: §3.1 · p. 10
  • A causal interpretation of conditional effects requires unconfoundedness, overlap, and no interference, which are assumptions rather than guarantees in observational data.These conditions cannot be tested directly from observed covariates, treatment, and outcome alone.Source: §3.1 · p. 10

4. Construction of a Pseudo-Treatment Variable

The study uses LogP to construct an exploratory pseudo-treatment in the observational BBBP dataset, rather than treating the resulting estimates as formal causal effects. A median split distinguishes higher- from lower-lipophilicity molecules.

4. Construction of a Pseudo-Treatment Variable

  • Because BBBP is observational and has no assigned treatment, the constructed treatment supports exploratory GRF analysis, not a formal causal-effect claim.The paper explicitly distinguishes these estimates from causal effects under the formal treatment-effect definition.Source: §4 · p. 10
  • LogP was selected as the pseudo-treatment basis because it quantifies lipophilicity, which is associated with passive BBB permeability.Moderate lipophilicity is described as generally favorable for permeability, while excessive LogP can limit bioavailability by reducing aqueous solubility.Source: §4 · pp. 10–11
  • A median split assigns molecules with LogP above the dataset median to the high-lipophilicity group and all others to the low group.The indicator is 1 above the median and 0 otherwise.Source: §4 · p. 11

5. Methodology

The GRF analysis uses molecular descriptors and SMILES bigrams as covariates, with LogP reserved for the pseudo-treatment. Honest forests produce molecule-level conditional-effect estimates, confidence intervals, and feature importances.

5. Methodology

  • LogP is excluded from the covariate matrix so it serves only as the pseudo-treatment and cannot leak into the predictors.The selected covariates are SMILES bigrams and molecular descriptors.Source: §5 · p. 11
  • The GRF estimates a conditional effect and asymptotic confidence interval for each molecule, alongside a normalized importance score for each covariate.The forest uses honest splitting, 2,000 trees, and a minimum leaf size of 10 observations.Source: §5 · p. 11
  • The preliminary analysis uses CausalForest, whereas the primary analysis uses CausalForestDML to incorporate orthogonalized nuisance estimation.Both implementations use the econml Python library.Source: §5 · p. 11
  • The procedure constructs a median-split LogP pseudo-treatment, removes LogP from X, and uses honest splitting with 2,000 trees and minimum leaf size 10.The algorithm also describes bootstrap sampling and separate structure and estimation subsets.Source: §5 · p. 12

6. Results and Discussion

The naive causal forest suggested substantial heterogeneity, but orthogonalized estimation attenuated it; after FDR correction, no conditional effects remained significant. Feature importance shifted from broad physicochemical descriptors toward SMILES bigrams, while the median split itself only balanced group sizes.

6.1. Treatment Group Balance

  • 815 (49.97%) high-LogP and 816 (50.03%) low-LogP molecules formed near-equal groups, but this alone establishes neither overlap nor unconfoundedness.The median LogP cutoff was 0.076; group balance was a mechanical consequence of the median split.Source: §6.1 · p. 13

6.2. Preliminary Estimates: Naive Causal Forest

  • Before adjustment, the naive forest estimated mean CATE 0.121 (SD 0.114; range [−0.101, 0.445]) across 1,631 molecules, with 46.9% significant at the 5% level.TPSA (importance 0.364) and heteroatom count (0.202) together accounted for over 56% of explained heterogeneity; the authors caution against interpreting this baseline as genuine heterogeneity.Source: §6.2 · p. 13

6.3. Orthogonalized Estimation via Double Machine Learning

  • Cross-fitted outcome and propensity models residualized permeability and treatment assignment on covariates before the residuals entered an honest causal forest.This procedure was used to address confounding and reduce regularization bias while preserving asymptotic consistency and valid inference.Source: §6.3 · pp. 13–14

6.3.1. Feature Importance

  • SMILES bigram sc was the largest individual contributor, and SMILES-derived features accounted for aggregate importance of 0.680 after orthogonalization.These importance scores explain variation in estimated treatment effects, not variables that directly determine BBB permeability.Source: Table 3 · p. 14, §6.3.1 · p. 14

6.3.2. Statistical Significance

  • After FDR correction, the orthogonalized forest had no significant effects (smallest adjusted p-value 0.082), versus 713 molecules (43.72%) for the naive forest.The naive model had 873 nominally significant molecules (53.53%); its CATE estimates were broadly spread, while orthogonalized estimates clustered near zero.Source: §6.3.2 · p. 15, Table 4 · p. 16

6.4. Summary of Heterogeneity Findings

  • After FDR correction, there was no evidence that LogP–BBB associations varied systematically across compounds in this dataset; apparent naive heterogeneity was attributed to descriptor confounding.The authors describe the relationship as largely homogeneous after accounting for other physicochemical properties, within this dataset's scope.Source: §6.4 · pp. 15–16

CRediT authorship contribution statement

The authors divide conceptualization, drafting, and technical work among three contributors, with two contributors supervising and reviewing the manuscript.

CRediT authorship contribution statement

  • Rapolai handled conceptualization, software, formal analysis, and the original draft; Makgai and Arashi contributed supervision, methodology, validation, and review.Rapolai also contributed methodology; Makgai and Arashi both supervised and validated the work.Source: CRediT authorship contribution statement · p. 17

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors used Claude and Gemini to improve manuscript language and readability, then reviewed and edited the content.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

  • Claude and Gemini were used to improve manuscript language and readability, after which the authors reviewed and edited the content.The authors state that they take full responsibility for the published article's content.Source: Declaration of generative AI and… · p. 17

Related papers

Loading 2609.29076v1…