Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization

Izhar Wallach; Abraham Heifets

Source-linked AI summary

Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization

Izhar Wallach, Abraham Heifets

arXiv:1706.06619v2Jun 20, 2017q-bio.QMcs.LGstat.ML

TL;DR

Ligand-based benchmarks may overestimate prospective accuracy when training-validation redundancy enables memorization. The paper extends MUV-style analysis into AVE, which includes inactive-molecule similarity, and finds AVE bias predicts performance across diverse benchmarks and modeling choices. The authors therefore interpret benchmark success as potentially reflecting overfitting rather than generalization, while noting limits of temporal splits and prospective scope.

Problem
Benchmark performance may be overly optimistic when training-validation redundancy allows memorization instead of generalization to novel data.
Method
The paper defines AVE, extends MUV-style functions to training and validation sets, and evaluates bias across ligand-based benchmarks and predictive methods.
Results
AVE bias predicts model performance across seven benchmarks, algorithms, fingerprints, distance metrics, and previously applied unbiasing techniques.
Takeaways & Limitations
High-bias tests can be easy to solve through relative active or inactive clustering, so benchmark performance may reward overfitting rather than generalization.
Takeaways & Limitations
Temporal splits omit molecules rejected before synthesis or assay, limiting how well they represent prospective discovery choices.

Abstract

from arXiv · show

Undetected overfitting can occur when there are significant redundancies between training and validation data. We describe AVE, a new measure of training-validation redundancy for ligand-based classification problems that accounts for the similarity amongst inactive molecules as well as active. We investigated seven widely-used benchmarks for virtual screening and classification, and show that the amount of AVE bias strongly correlates with the performance of ligand-based predictive methods irrespective of the predicted property, chemical fingerprint, similarity measure, or previously-applied unbiasing techniques. Therefore, it may be that the previously-reported performance of most ligand-based methods can be explained by overfitting to benchmarks rather than good prospective accuracy.

Introduction

Ligand-based benchmarks can overstate predictive performance when training-validation redundancy lets models memorize chemical patterns rather than generalize to novel data. The paper introduces a bias measure motivated by failures of existing unbiasing approaches and evaluates its relationship with benchmark accuracy.

Motivation: Benchmark accuracies often exceed 0.8 AUC, but biochemical gold-standard predictions remain measured rather than computed.The authors therefore question whether benchmark design yields overly optimistic performance estimates.
Motivation: When training and validation data coincide, memorization can produce perfect benchmark performance while failing completely on new data.Such performance reflects benchmark-design failure rather than successful prediction.
Motivation: Benchmark validation should reflect generalization to novel data of the kind evaluated systems would encounter in practice.MNIST and TIMIT illustrate this principle by splitting data according to authors or speakers rather than randomly by example.
Chemical-data bias: Chemical datasets contain redundancies because synthesized molecules and medicinal-chemistry efforts sample chemical space unevenly and cluster near promising molecules.These patterns can dominate benchmark measurements without indicating predictive value for novel investigations.
Prior bias controls: Existing bias sources include analog bias, artificial enrichment, and complexity bias, where scaffold or coarse-property differences can make actives easy to distinguish.Several clustering, temporal, and similarity-balancing techniques had already been proposed to mitigate redundancy.
Paper contribution: AVE measures whether 1-NN can solve validation data by memorizing training data and predicts performance across algorithms, features, metrics, benchmarks, and unbiasing techniques.The paper also applies a bias-reduction algorithm and reports native and unbiased benchmark performance.

Bias Measurement Method

The method extends MUV distribution functions to compare validation and training molecules across classes and similarity thresholds. AVE combines active and inactive relative-clumping terms, so bias captures redundancy exploitable in either class.

Function construction: The method extends MUV nearest-neighbor and empty-space functions from a single validation set to paired training and validation sets.This adaptation makes the distribution analysis applicable to machine-learning comparisons whose behavior depends on training data.
Illustration: Figure 1 shows that equal AVE bias can predict similarly good performance for datasets with high or low overall training-validation similarity.Relative similarity between same-class and opposite-class molecules can support a separating hyperplane even when validation molecules appear dissimilar overall.
Function construction: S(V,T,d) evaluates whether validation molecules have nearby neighbors in a training set under similarity threshold d.The supplied definition identifies V as validation molecules, T as training molecules, and uses an indicator for the nearest-neighbor distance condition.
AVE definition: AVE compares validation actives and inactives with training actives and inactives across distance thresholds D.Va, Vi, Ta, and Ti denote validation actives, validation inactives, training actives, and training inactives.
AVE interpretation: Bias decreases as B approaches zero, indicating a less biased benchmark.The measure therefore treats lower absolute redundancy bias as preferable.
AVE definition: The bias B combines active clumping, AA − AI, with inactive clumping, II − IA.The inactive term extends MUV by measuring patterns among inactives because classifiers can learn structure in either class.

Results

Across ligand-based benchmarks, AVE bias consistently tracks apparent classification performance across algorithms, fingerprints, partitioning schemes, and chemical properties. These results indicate that benchmark redundancy can make models appear accurate through memorization rather than generalization.

J&J Benchmark: AVE bias correlated with AUC across four classifiers on the J&J benchmark, where average AUCs ranged from 0.76 to 0.85.RF, SVM, LR, and KNN achieved average AUCs of 0.85, 0.81, 0.84, and 0.76, respectively.
J&J Benchmark: Alternative fingerprints preserved the bias-performance relationship on J&J, including MACCS and simple atom-count and molecular-property features.For MACCS, RF achieved AUC 0.82 with ρ = 0.82, τ = 0.73, and r2 = 0.68; simple fingerprints also produced high AUCs.
MUV Benchmark: MUV showed high AVE-AUC correlations, with r2 ranging from 0.73 to 0.88 across classifiers.The active-similarity component was negative, while the inactive-similarity component was approximately 0.4 across targets, making inactive classification generally easier.
MUV Benchmark: MUV projections revealed clustering among decoys and separation from actives, allowing random train-validation splits to support memorization of inactive molecules.The observed structure could permit good performance even when active selection was designed to be evenly distributed against the inactive background.
Cross-benchmark analysis: Cross-target transfer tests produced modest above-random AUCs, while high AVE bias identified prediction pairs where most algorithms exceeded AUC 0.9.Across 90 unrelated-target prediction pairs, average AUCs were 0.59 for RF, 0.60 for SVM, 0.56 for LR, and 0.52 for 1-NN.

Discussion

The discussion argues that benchmark redundancy can make strong held-out performance reflect memorization rather than generalization, especially in cheminformatics settings where discovery data are nonstationary and biased. AVE analysis exposes this problem across benchmarks and motivates careful, feature-aware validation and cautious interpretation of prospective accuracy.

Scope and motivation: Cheminformatics requires predictions for examples unlike training data because biochemical discovery draws from nonstationary distributions.New chemical matter and unexplored target classes make this mismatch particularly challenging for machine learning.
Benchmark design: Existing debiasing methods often focus on active-active similarity while assuming diverse inactives, overlooking inactive-inactive redundancy and absolute similarity thresholds.The paper identifies scaffold clustering, fingerprint clustering, temporal splitting, and maximum unbiased validation as examples of these approaches.
Main finding: Across seven previously unbiased benchmarks, AVE bias showed substantial r2 correlations with AUC for every tested machine-learning algorithm.The authors propose adding AVE bias analysis to assess redundancy among inactive molecules and identify performance patterns consistent with overfitting.
Main finding: In the shuffled Merck experiment, models trained on one target predicted another without seeing any active for the intended target, yet RF achieved r2 = 0.73, Pearson’s ρ = 0.86, and Kendall’s τ = 0.61.The authors attribute the usable signal to training inactives and report that AVE bias identified tests on which models would perform well or fail.
Main finding: The 1-NN results indicate overfitting in every investigated benchmark, while AVE bias predicts when memorization can solve a benchmark through training-validation similarity.On J&J ECFP4, RF reached AUC 0.85 versus 1-NN AUC 0.76, showing that the validation data were not inherently too complex for other models.
Practical implications: The proposed genetic-algorithm partitioning can reduce AVE bias when data are scarce, but it is heuristic, not guaranteed optimal, and may introduce new biases.Fixed-distance exclusion can instead remove too much data, including difficult points near Voronoi-cell borders.

Conclusion & Recommendations

The paper introduces AVE bias to assess training-validation split difficulty and shows that benchmark redundancy predicts apparent machine-learning performance. It recommends measuring and reporting benchmark bias, using baselines, and prioritizing prospective discovery tests.

Conclusion & Recommendations: AVE bias evaluates the difficulty of training-validation data splits, addressing overfitting risks that previous unbiasing techniques did not fully consider.The measure accounts for machine-learning methods learning from both active and inactive classes.
Conclusion & Recommendations: High-AVE-bias tests are easy to solve, and AVE bias predicts performance across benchmarks, algorithms, and molecular fingerprints.This relationship remains after clustering, approximate temporal splitting, and active-active distance balancing have been applied.
Conclusion & Recommendations: Relative clustering of actives and inactives can make test classes separable even when training and validation molecules appear different.Machine-learning algorithms can exploit similarity among either actives or inactives to find a separating hyperplane.
Conclusion & Recommendations: Removing AVE bias eliminates much of the performance of modern machine-learning approaches.This supports the interpretation that measured performance depends substantially on benchmark redundancy.
Conclusion & Recommendations: The results are consistent with benchmark memorization of inactive-molecule similarity explaining apparent gains in ligand-based screening.The interpretation also accounts for correlations among fingerprints and machine-learning algorithms, alongside high benchmark performance and low real-world utilization.
Conclusion & Recommendations: The authors recommend measuring and reporting AVE bias, excluding or unbiasing highly biased tests, including baseline performance, and using prospective discovery testing.Structure-based technologies can model prospective testing by withholding all actives for a target while avoiding contamination from closely related targets.