PaperlayerView exact revision on arXiv

Source-linked AI summary

ToxLens: A Reproducible Graph-Learning Framework for Leakage-Aware, Uncertainty-Calibrated Molecular Toxicity Prediction

Magnus H. Strømme, Alex G. C. de Sá, David B. Ascher

arXiv:2608.30472v1cs.LG

TL;DR

Molecular toxicity benchmarks may overstate practical reliability when related structures cross training and test folds, while toxicity triage also needs uncertainty and interpretable structural evidence. ToxLens addresses these needs with a leakage-aware, multi-task graph-learning workflow and achieves stronger performance than the specified shallow baselines on its controlled split, while external results remain competitive but not uniformly state of the art.

  • Problem

    Structurally related molecules across training and test folds can inflate benchmark performance, motivating evaluation that also considers leakage, uncertainty, applicability domain, and interpretability.

  • Method

    ToxLens combines conservative curation, leakage-aware partitioning, graph and global-feature encoders, calibrated Monte Carlo-dropout prediction sets, applicability-domain analysis, and SHAP-guided occlusion.

  • Results

    On the leakage-controlled 11-endpoint panel, ToxLens achieved macro MCC 0.44, macro AUROC 0.83, and macro AUPRC 0.58, exceeding four ECFP4-based shallow baselines on all endpoints.

  • Takeaways & Limitations

    The evidence supports ToxLens as a reproducible decision-support framework, with strongest support for shallow-baseline comparison on the fixed in-house split.

  • Takeaways & Limitations

    Conclusions remain constrained by one in-house partition, limited external-validity evidence, non-strict split-conformal guarantees, and unresolved limits on mechanistic interpretation.

Abstract

from arXiv · show

Molecular toxicity prediction is increasingly used to prioritise compounds before experimental testing, but conventional benchmark performance can overstate practical utility when structurally related molecules occur across training and test folds. We introduce ToxLens, a reproducible multi-task graph-learning framework for 11 toxicity endpoints spanning Ames mutagenicity, acute oral toxicity, hERG inhibition, and Tox21 nuclear-receptor and stress-response assays. The workflow combines conservative chemical curation, sphere-exclusion filtering, a leakage-aware UMAP-HDBSCAN split, parallel graph and global-feature encoders joined by late concatenation, temperature-scaled Monte Carlo dropout with conformal-style prediction sets, applicability-domain analysis, and SHAP-guided toxicophore discovery with occlusion controls. On the leakage-controlled test fold, a five-seed soft-voting ensemble achieved a Matthews correlation coefficient score of 0.44, an area under the receiver operating characteristic curve score of 0.83, and an area under the precision-recall curve score of 0.58. It exceeded four ECFP4-based shallow baselines on all 11 endpoints under the same split and validation-based threshold-selection protocol. Controlled ablations showed that the global pathway was important, whereas late concatenation outperformed the tested gated and feature-wise linear modulation fusion variants. Conformal-style prediction sets revealed substantial endpoint-specific variation in set efficiency, and discrimination and calibration improved with similarity to the training domain. Retraining on fixed published Tox21 Challenge and TDA folds produced competitive, but not uniformly state-of-the-art, performance. SHAP-guided occlusion and consensus subgraph mining yielded model-derived structural hypotheses, 44 of which contained at least one occurrence that passed the predefined counterfactual criteria.

1. Introduction

ToxLens targets practical reliability in molecular toxicity prediction by combining leakage-aware evaluation, uncertainty reporting, applicability-domain analysis, and chemically grounded interpretation. It evaluates this framework across 11 toxicity endpoints and reports stronger performance than the specified shallow baselines on the leakage-controlled split.

  • Motivation: Benchmark metrics can be inflated by structurally related molecules across folds, motivating structural separation and applicability-domain reporting alongside discrimination.
  • Motivation: Forced binary decisions are poorly matched to out-of-domain, sparse, noisy, or imbalanced compounds that may warrant additional testing.Conformal prediction instead represents uncertainty through prediction sets, including ambiguous two-class outputs.
  • Framework: ToxLens integrates chemical curation, sphere exclusion, UMAP-HDBSCAN partitioning, graph and global-feature pathways, conformal-style uncertainty, and SHAP-guided toxicophore mining.The workflow preserves strict separation of training, validation, and test uses and supports endpoint-level reportability auditing.
  • Results: Macro MCC 0.44, macro AUROC 0.83, and macro AUPRC 0.58 were achieved by the five-seed ensemble on the leakage-controlled split.The ensemble exceeded four ECFP4-based shallow baselines on all 11 endpoints under the same split and validation-threshold protocol.
  • Framework: 11 endpoints span Ames mutagenicity, acute oral toxicity, hERG inhibition, and Tox21 nuclear-receptor and stress-response assays.The model uses hard parameter sharing with task-specific output layers across these endpoint heads.
  • Interpretation: SHAP-guided occlusion evaluates whether consensus fragments produce larger prediction drops than within-molecule random subgraph controls.Passing motifs are model-derived structural hypotheses rather than chemical deletions, causal interventions, or experimental validation.

3. Software and Reproducibility Resources

The reproducibility package provides the materials needed to reproduce the reported analyses without a public prediction server.

  • The package includes source code, environment specifications, exact package locks, fixed data splits, checkpoints, provenance records, and machine-readable results.

4. Results

ToxLens performed strongly under leakage-controlled evaluation, with ensemble gains over shallow baselines and endpoint-dependent uncertainty and applicability-domain behavior. Ablations and external benchmarks support a narrower conclusion: global features and late concatenation were useful, while external performance was competitive but not uniformly state-of-the-art.

  • Ensemble and shallow-baseline comparison: The five-seed ensemble achieved macro test MCC 0.44, macro AUROC 0.83, and macro AUPRC 0.58 across 11 endpoints.It achieved the highest test MCC on all 11 endpoints against the strongest endpoint-specific ECFP4 shallow comparator.
  • Ensemble and shallow-baseline comparison: ToxLens achieved the highest test MCC on all 11 endpoints against endpoint-specific ECFP4 shallow baselines trained under the same split and threshold protocol.The strongest shallow comparator varied by endpoint, and the largest margins occurred for SR-MMP, SR-HSE, NR-ER, and SR-p53.
  • Architectural ablation: Removing the global pathway produced the weakest tested configuration, with A3 reaching test MCC 0.341 and AUROC 0.791.This supports a contribution from global features beyond the graph representation.
  • Architectural ablation: Under the fixed ablation protocol, no-mid-trunk fusion achieved higher test MCC than the tested GCMI and FiLM variants, supporting late concatenation as the default.The controlled conclusion does not establish the proposed mechanism as the cause, and the ablations were single-run comparisons.
  • Uncertainty and applicability domain: Singleton prediction-set efficiency varied markedly by endpoint, exceeding 80% for five endpoints and reaching 100% for NR-ER-LBD.Two-label sets indicate that both classes meet the endpoint-specific calibration threshold and are flagged for further review.
  • Uncertainty and applicability domain: Mean AUROC increased from 0.77 in the most distant similarity quartile to 0.86 in the most similar, while mean MCC rose from 0.28 to 0.44 and mean expected calibration error fell from 0.13 to 0.08.This is a panel-level association; individual endpoints were not uniformly monotonic.
  • External benchmark evaluation: On the canonical Tox21 Challenge partition, ToxLens achieved macro AUROC 0.82 across 12 endpoints, below the published DeepTox DNN-only mean of 0.84 and winning ensemble mean of 0.85.The experiment primarily reproduced the published protocol with modern uncertainty and interpretability outputs rather than establishing state-of-the-art performance.
  • External benchmark evaluation: Across fixed TDC ADMET folds, ToxLens was competitive with reference architectures but did not uniformly exceed their leaderboard values.These comparisons support transfer to independently curated folds, not prediction of unseen endpoints or a universal state-of-the-art claim.

5. Discussion

ToxLens combines predictive modelling with leakage auditing, uncertainty reporting, applicability-domain analysis, and model-level interpretation. The evidence is strongest for the reported shallow-baseline comparison on the fixed in-house split, while external superiority and mechanistic claims remain limited.

  • ToxLens integrates leakage-controlled validation, calibrated abstention, applicability-domain analysis, and chemically grounded interpretation into one auditable workflow.
  • 0.44 macro MCC, 0.83 macro AUROC, and 0.58 macro AUPRC were achieved on the leakage-controlled 11-endpoint panel.The ensemble exceeded four ECFP4-based shallow baselines on all endpoints under the same split and validation-threshold protocol.
  • Removing the global pathway produced the weakest tested configuration, while GCMI and FiLM did not improve on late concatenation.The proposed explanation involving suppressed complementary graph information remains a hypothesis because information loss was not directly measured.
  • Prediction-set efficiency varied substantially by endpoint, while validation reuse prevents treating α = 0.05 as an exact finite-sample coverage guarantee.Two-label sets flag compounds for further review, but prospective calibration drift was not evaluated.
  • Higher similarity to the training set was associated with better aggregate discrimination and calibration, but similarity is not a deterministic acceptance rule.The trend was not monotonic for every endpoint, particularly where quartile-specific positive counts were small.
  • External retraining was competitive on selected fixed benchmarks but did not establish universal state-of-the-art performance or transfer to unseen endpoints.The interpretation workflow supports model-level structural hypotheses, not demonstrated biological mechanisms; consensus motifs require independent review and experimental validation.

6. Limitations

The conclusions are constrained by split variability, calibration reuse, limited external validity, model-level interpretation, and an absent PubChem feature block.

  • One UMAP-HDBSCAN partition measures optimisation variability across five seeds but not variability across alternative chemistry-aware splits.
  • External benchmarks test protocol-level transfer rather than external validity of the fitted 11-endpoint checkpoint and do not uniformly exceed reported reference values.
  • Validation-fold reuse for checkpoint selection, temperature scaling, and calibration means the exact finite-sample split-conformal guarantee does not apply.
  • SHAP, occlusion, and consensus clustering provide model-level analyses without prospective experimental validation, while some motifs are short or generic.
  • The reported checkpoint had a zero 200-dimensional PubChem block because its cache was absent, so conclusions about PubChem bioactivity features are unavailable.

7. Future Directions

Future work targets stronger evaluation under chemistry-aware and deployment shifts, experimental testing of model-derived motifs, and better ensemble comparability.

  • Repeated chemistry-aware splits and prospective temporal or external cohorts should test robustness beyond the single present partition.
  • Conformal prediction should be recalibrated under measured deployment shift to assess prediction-set behaviour prospectively.
  • Model-derived motifs should be tested experimentally rather than treated as prospectively validated toxicophores.
  • A multi-seed Tox21 Challenge evaluation would reduce the present ensemble-depth asymmetry with published DeepTox results.
  • Assay-condition metadata may support larger multi-assay models, but negative transfer should be evaluated rather than assumed absent.

8. Conclusion

ToxLens combines leakage-aware, uncertainty-aware, and interpretable toxicity modelling components with reproducible benchmarking. Its in-house ensemble outperformed ECFP4 shallow baselines across all endpoints, while external results remained competitive but not uniformly state of the art.

  • ToxLens integrates an 11-task graph-plus-global classifier with leakage-aware partitioning, conformal-style sets, applicability-domain analysis, external retraining, and controlled interpretation.
  • Macro MCC 0.44, macro AUROC 0.83, and macro AUPRC 0.58 were achieved by the five-seed ensemble on the in-house held-out split.
  • The ensemble exceeded the reported ECFP4 shallow baselines on all endpoints under the in-house evaluation.
  • External results were competitive but not uniformly state of the art, limiting claims of broad benchmark dominance.
  • Supplementary materials document per-task counts, thresholds, conformal quantiles, baseline comparisons, partition strategies, applicability-domain performance, and ranking metrics.
  • The endpoint data came from Therapeutics Data Commons toxicity tasks and the published hERG_Karim dataset, with the external Tox21 Challenge using the canonical DeepTox archive.

1. Supplementary Data

Supplementary analyses compare structural separation across partitioning strategies and audit endpoint imbalance, weighting choices, and final split counts.

  • Nearest-neighbour Tanimoto distributions compare Random, Butina, Bemis-Murcko scaffold, and UMAP-HDBSCAN partitions, with lower values indicating stronger structural separation.
  • Supplementary Table 3 complements similarity distributions with scaffold novelty, property balance, and aggregate out-of-distribution statistics.
  • NR-Aromatase was the most imbalanced endpoint with 194 positives and 3,969 negatives, whereas Ames had 2,508 positives and 2,319 negatives.
  • Supplementary Table 1 reports training and validation counts, decision thresholds, and conformal quantiles after duplicate resolution and the final UMAP-HDBSCAN split.
  • The retained checkpoint used class-weight power zero, reducing both class weights to one and fitting unweighted binary cross-entropy with fixed endpoint-emphasis weights.

2. Supplementary Methods

ToxLens uses masked multi-task BCE with fixed task emphasis, validation-based thresholding, and a late-concatenation architecture rather than the tested GCMI comparator. Its uncertainty procedure combines temperature-scaled Monte Carlo dropout with endpoint-wise conformal-style calibration.

  • Model configuration: The retained model uses five GINE layers and fixed dropout, optimisation, stochastic-depth, and label-smoothing settings selected through training and validation data.The active configuration includes hidden width 256, learning rate 3 × 10⁻⁴, and label-smoothing coefficient 0.0007.
  • Threshold calibration: Per-task thresholds are selected on validation probabilities by maximising MCC over a 0.05–0.95 grid and then applied unchanged to the test fold.The procedure uses increments of 0.01 and does not alter the test-time threshold after selection.
  • Fusion architecture: Late concatenation was retained because it achieved higher test MCC than the ablated GCMI fusion mechanism.GCMI reweights node channels using global representations and adds a nonlinear synergy term, while retaining a direct structural path.
  • Classification loss: The masked objective averages BCE across tasks with valid labels, using fixed endpoint-emphasis weights and no learnable per-task scalars.Selected primary tasks receive weights from 1.10 to 1.35, while auxiliary losses have zero weight in the retained checkpoint.
  • Uncertainty estimation: Conformal-style calibration targets α = 0.05 using 30 stochastic dropout passes, temperature-scaled predictive means, and endpoint-specific nonconformity scores.Prediction sets include class 0 or 1 according to calibrated quantiles, with two-label sets flagged as uncertain.

3. Supplementary Results

Supplementary evaluations examine shallow ECFP4 baselines, alternative partition strategies, applicability-domain strata, Tox21 metrics, and reporting conventions. The held-out performance summaries are organised around leakage-aware splitting, similarity to the training domain, and ranking quality.

  • Baseline comparison: ECFP4 shallow baselines share the UMAP-HDBSCAN split and validation-MCC threshold calibration with the five-seed ToxLens ensemble across eleven endpoints.The baselines are Random Forest, XGBoost, MLP, and SVM using 1024-bit ECFP4 fingerprints.
  • Data partitioning: The partition-strategy comparison reports random, Butina, Bemis–Murcko scaffold, and UMAP-HDBSCAN splits using similarity, scaffold novelty, property balance, and OOD utility measures.Median nearest-neighbour Tanimoto similarity is interpreted as lower for stronger separation.
  • Applicability domain: Mean AUROC rises 0.77 → 0.82 → 0.82 → 0.86 and mean MCC rises 0.28 → 0.35 → 0.41 → 0.44 from the most distant to most similar applicability-domain quartiles.Each quartile contains approximately 742–748 molecules per task on the 2,970-molecule held-out test split.
  • Tox21 evaluation: The extended Tox21 supplementary table reports final-test per-task metrics including EF@1% and BEDROC for a single-seed evaluation.Scores use the canonical DeepTox tox21_labels_test.csv.gz partition and validation-optimal probability thresholds.
  • Reporting conventions: Confidence intervals are computed with 1,000-iteration percentile bootstrap, and reportable-primary rows are sorted by descending AUROC.Excluded endpoints are appended after the reportable-primary set.

Endpoint AUROC AUPRC MCC EF@1% BEDROC

Endpoint-level evaluation reports ranking and classification comparisons across ToxLens and published TDC references. The supplied results show endpoint-specific AUROC differences rather than uniform dominance over every comparator.

  • Ranking quality: The ranking-quality table evaluates BEDROC at α = 20 and enrichment factors at top-k percentages on the UMAP-HDBSCAN hold-out fold.EF@k% denotes enrichment at the top k% of ranked predictions.
  • External benchmark protocol: TDC retraining reports mean ± SD AUROC across five 90:10 development re-splits, with evaluation on untouched benchmark test folds and explicit overlap indicators.Comparator AUROCs are leaderboard point estimates, and clean_claim is false where inherent development/test overlap exists.
  • AMES: ToxLens trails Chemprop-RDKit by 0.019 on AMES AUROC, trails CMPNN by 0.012, and exceeds AttentiveFP by 0.017.These are per-comparator point-estimate deltas defined as ToxLens − TDC reference.
  • DILI: ToxLens trails Chemprop by 0.012 on DILI AUROC, matches Chemprop-RDKit at −0.000, and exceeds AttentiveFP by 0.001.The deltas are point estimates against the corresponding TDC references.
  • hERG: ToxLens trails DeepPurpose, Chemprop-RDKit, and AttentiveFP on hERG AUROC by 0.036, 0.035, and 0.020, respectively.All three values are point-estimate deltas relative to TDC references.
Loading 2608.30472v1…