PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences

Martin Buttenschoen; Garrett M. Morris; Charlotte M. Deane

Source-linked AI summary

PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences

Martin Buttenschoen, Garrett M. Morris, Charlotte M. Deane

arXiv:2308.05777v3Aug 10, 2023q-bio.QMphysics.chem-ph

TL;DR

Deep-learning docking methods can achieve favorable RMSD while producing physically implausible poses, making RMSD alone insufficient for evaluation. The paper introduces PoseBusters, an RDKit-based quality-check suite, and finds that no tested deep-learning method outperforms classical docking when plausibility and generalization are considered.

Problem
Deep-learning docking methods are often evaluated by RMSD despite producing physically implausible structures, so steric and energetic criteria are also needed.
Method
PoseBusters applies chemical, physical, and steric quality checks to poses from five deep-learning and two classical docking methods.
Results
No tested deep-learning docking method outperforms standard docking when physical plausibility and binding-mode RMSD are considered; force-field minimization improves poses but does not reverse this comparison.
Takeaways & Limitations
PoseBusters provides a PB-validity criterion that can expose docking-relevant failure modes and guide development of improved inductive biases.
Takeaways & Limitations
Time-based train-test splits are insufficient to assess generalization to novel targets, so protein sequence identity between training and test sets must be reported.

Abstract

from arXiv · show

The last few years have seen the development of numerous deep learning-based protein-ligand docking methods. They offer huge promise in terms of speed and accuracy. However, despite claims of state-of-the-art performance in terms of crystallographic root-mean-square deviation (RMSD), upon closer inspection, it has become apparent that they often produce physically implausible molecular structures. It is therefore not sufficient to evaluate these methods solely by RMSD to a native binding mode. It is vital, particularly for deep learning-based methods, that they are also evaluated on steric and energetic criteria. We present PoseBusters, a Python package that performs a series of standard quality checks using the well-established cheminformatics toolkit RDKit. Only methods that both pass these checks and predict native-like binding modes should be classed as having "state-of-the-art" performance. We use PoseBusters to compare five deep learning-based docking methods (DeepDock, DiffDock, EquiBind, TankBind, and Uni-Mol) and two well-established standard docking methods (AutoDock Vina and CCDC Gold) with and without an additional post-prediction energy minimisation step using a molecular mechanics force field. We show that both in terms of physical plausibility and the ability to generalise to examples that are distinct from the training data, no deep learning-based method yet outperforms classical docking tools. In addition, we find that molecular mechanics force fields contain docking-relevant physics missing from deep-learning methods. PoseBusters allows practitioners to assess docking and molecular generation methods and may inspire new inductive biases still required to improve deep learning-based methods, which will help drive the development of more accurate and more realistic predictions.

1 Introduction

Docking supports structure-based drug discovery, but RMSD-only evaluations can overlook chemically and physically implausible poses from deep-learning methods. PoseBusters is introduced to evaluate plausibility alongside native-like binding modes across deep-learning and classical docking methods.

Docking predicts predominant protein–ligand binding modes for virtual screening and medicinal-chemistry interpretation.
Deep-learning docking methods promise fast and accurate virtual screening but may lack inductive biases that enforce chemical consistency and physical plausibility.
RMSD-only comparisons dominate docking evaluation, although recent deep-learning methods’ performance claims rely entirely on RMSD and omit physical-plausibility testing.
PoseBusters evaluates poses from five deep-learning methods and two standard methods on the Astex Diverse and PoseBusters Benchmark sets.
On the Astex Diverse set, DiffDock performs best by RMSD alone, whereas Gold and AutoDock Vina perform best when physical plausibility is considered.

2 Methods

The study re-docks known ligands with five diverse deep-learning methods and two established classical methods, using their published models and defined search spaces. The methods are compared to assess accurate, chemically valid, and physically valid pose prediction.

The evaluation re-docks known ligands into their respective proteins and assesses predicted poses with PoseBusters.
The comparison includes five methodologically diverse deep-learning docking methods and the classical methods AutoDock Vina and Gold.
DeepDock, DiffDock, EquiBind, TankBind, and Uni-Mol use distinct neural architectures and docking strategies, including blind docking and pocket-based prediction.
All five deep-learning methods were trained on subsets of the PDBbind General Set, with DeepDock using v2019 and the other four using v2020.
DeepDock and Uni-Mol require binding-site definitions, whereas DiffDock, EquiBind, and TankBind search over the entire protein as blind docking methods.

2.2 The PoseBusters test suite

PoseBusters combines chemical-validity, intramolecular, and intermolecular tests to identify chemically inconsistent, energetically unlikely, geometrically abnormal, or clashing ligand poses. Its checks use RDKit-derived molecular representations, geometry bounds, force-field energies, and molecular-overlap criteria.

PoseBusters organizes checks into chemical validity, intramolecular properties, and intermolecular interactions.
Chemical validity and consistency: Chemical checks test RDKit sanitization and consistency between predicted and true ligands using standardized InChI layers.
Intramolecular validity: Intramolecular checks validate bond lengths, bond angles, internal distances, aromatic or double-bond flatness, and ligand-conformation energy.
Intramolecular validity: The energy test rejects docked conformations whose energy exceeds a user-specified ratio relative to 50 generated, UFF-relaxed conformations.
Intermolecular interactions: Intermolecular checks assess minimum heavy-atom distances and overlapping van der Waals volumes between ligands, proteins, and cofactors.
Intermolecular interactions: The default intermolecular criteria use a 0.75 distance ratio threshold and a 7.5% volume-overlap threshold, with configurable radii scaling.

2.3 Quality of fit

PoseBusters measures quality of fit using symmetry-aware heavy-atom RMSD between a predicted ligand pose and the closest crystallographic ligand. Coverage records the share of predictions within a default 2 Å RMSD threshold.

PoseBusters computes minimum symmetry-aware heavy-atom RMSD between the predicted pose and the closest crystallographic ligand.
Coverage is the share of predictions within a user-adjustable RMSD threshold, defaulting to 2 Å for regular-size ligands.

2.4 Sequence identity

Sequence identity is defined as the proportion of exact amino-acid residue matches after aligning two protein chains.

Sequence identity equals exact residue matches after alignment divided by the query sequence length.Alignments use Smith–Waterman with specified gap scores and BLOSUM62; unknown residues count as mismatches.

2.5 Molecular mechanics energy minimisation

Post-docking molecular mechanics minimisation optimises ligand coordinates in the binding pocket while keeping protein atoms fixed.

Energy minimisation uses AMBER ff14sb and Sage force fields in OpenMM, updating only ligand atom positions.Protein structures are prepared with PDBfixer and minimisation continues until energy convergence within 0.01 kJmol−1.

2.6 Data

The study uses the established Astex Diverse set and a new PoseBusters Benchmark set to evaluate docking across familiar and unseen complexes.

2.6.1 Astex Diverse set: The Astex Diverse set contains hand-picked, relevant, diverse, high-quality protein-ligand complexes from the PDB.The complexes were processed into protein PDB files and ligand SDF files after solvent and ligand removal.
2.6.2 PoseBusters Benchmark set: The PoseBusters Benchmark set contains 308 recent, diverse, high-quality crystal complexes with drug-like molecules released since 2021.Its release-date restriction excludes complexes from the PDBBind General Set v2020 used to train many evaluated methods.
2.6.2 PoseBusters Benchmark set: The Astex Diverse set contains 85 cases, whereas the PoseBusters Benchmark set contains 308 cases designed to test generalisation beyond training complexes.Predictions are assessed by RMSD within 2 Å and, separately, by passing all PoseBusters geometry and energy checks.

3 Results

PoseBusters shows that RMSD alone can overstate docking quality: classical methods generally produce more physically valid and generalisable poses than deep-learning methods, although minimisation repairs many DL predictions.

3.1 Results on the Astex Diverse set: For TankBind on Astex, 50 of 85 predictions have RMSD within 2 Å, but only 5 pass all PoseBusters tests.The waterfall plot shows successive physical-plausibility filters removing predictions.
3.1 Results on the Astex Diverse set: On Astex, DiffDock reaches 72% RMSD coverage, but Gold leads PB-valid performance at 64% after physical-plausibility filtering.AutoDock Vina reaches 56% and DiffDock 47% PB-valid coverage; DiffDock is the only DL method comparable to standard methods there.
3.1 Results on the Astex Diverse set: DL methods fail distinct physical checks: TankBind often violates stereochemistry, Uni-Mol bond lengths, and EquiBind protein-ligand clash criteria.Gold and AutoDock Vina pass most tests but still produce some protein-ligand clashes.
3.2 Results on the PoseBusters Benchmark set: On the unseen PoseBusters Benchmark, Gold and AutoDock Vina perform best, while DiffDock reaches only 12% as the best DL method.All methods perform worse on this benchmark than on Astex, and EquiBind, Uni-Mol, and TankBind produce almost no fully valid poses.
3.2 Results on the PoseBusters Benchmark set: DL methods perform far better on test proteins with high sequence identity to training proteins than on more novel sequences.Sequence identity is stratified relative to the PDBBind General Set v2020 to approximate similarity to training targets.
3.3 Results with pose-docking energy minimisation: Post-docking energy minimisation significantly increases physically plausible predictions for DiffDock, DeepDock, TankBind, and Uni-Mol.This repair effect indicates that docking-relevant force-field physics is missing from at least some DL methods; AutoDock Vina and Gold do not improve.

4 Discussion

PoseBusters evaluates docking predictions using physical plausibility alongside binding-mode RMSD, revealing that deep learning methods do not yet outperform classical docking tools. The analyses also expose poor generalisation to novel sequences and show that force-field minimisation can improve predictions without closing this gap.

Evaluation and physical plausibility: PoseBusters identified chemically inconsistent and physically implausible poses across five deep learning and two standard docking methods.The evaluated methods were DeepDock, DiffDock, EquiBind, TankBind, Uni-Mol, AutoDock Vina, and Gold.
Evaluation and physical plausibility: No deep learning docking method outperformed standard docking methods when physical plausibility and binding-mode RMSD were considered together.This demonstrates why RMSD alone is insufficient for evaluating docking predictions.
Failure modes: Individual PoseBusters tests revealed distinct failure modes, including non-standard bond lengths in Uni-Mol and internal ligand clashes in TankBind.These tests can help developers identify inductive biases for improving binding-mode prediction methods.
Generalisation: Deep learning docking methods generalised poorly to novel data and were prone to overfitting to the proteins in their training data.Performance was poor overall on the PoseBusters Benchmark set, especially for targets with low sequence identity to PDBbind 2020.
Generalisation: For targets with sequence identity below 30% to any PDBbind General Set v2020 member, almost no deep learning predictions were physically valid within the 2 Å threshold.This result directly links low sequence identity with a scarcity of predictions satisfying both physical validity and the RMSD criterion.
Energy minimisation: Force-field energy minimisation considerably improved deep learning docking poses, but DiffDock still did not outperform Gold or AutoDock Vina.The results indicate that docking-relevant chemistry and physics encoded in force fields are missing from at least some deep learning models.
Implications: PoseBusters adds PB-validity beyond the traditional RMSD ≤2 Å rule to support evaluation of chemical consistency, physical plausibility, and generalisability.The authors propose that future methods should aim to outperform standard tools on both RMSD and these additional criteria.

Data availability

PoseBusters is distributed as an installable open-source Python package, with the paper’s datasets and tabulated docking results deposited separately.

Software: PoseBusters is available as a pip-installable Python package under the BSD-3-Clause license.The source code is hosted at github.com/maabuu/posebusters.
Data: The Astex Diverse and PoseBusters Benchmark datasets are available through Zenodo.The cited Zenodo record is 8278563.
Data: Zenodo also provides the individual tabulated test results for each docking method.These materials accompany the paper’s datasets.

Conflicts of interest

The paper declares no conflicts of interest.

Declaration: The authors report that there are no conflicts to declare.

PoseBusters Supplementary information

The supplementary information documents authorship, affiliations, docking protocols, benchmark construction, validation assumptions, and supplementary comparisons for PoseBusters.

Publication information: The paper is authored by Martin Buttenschoen, Garrett M. Morris, and Charlotte M. Deane at the University of Oxford’s Department of Statistics.The supplementary material identifies the Oxford address as 24-29 St Giles’, Oxford OX1 3LB, United Kingdom.
Docking protocols: The supplementary protocols re-dock ligands from the Astex Diverse and PoseBusters Benchmark sets into crystal structures using seven docking methods.Methods requiring an initial ligand conformation received identical RDKit ETKDGv3 conformers followed by UFF minimisation.
AutoDock Vina: AutoDock Vina generated 40 poses with exhaustiveness 32 and selected the top-ranked pose within a 25 Å bounding box around the crystal-ligand centroid.
Gold: Gold generated 40 poses in a 25 Å-radius binding site around the crystal-ligand centroid and retained the top-ranked pose.The settings used PLP rescoring, 100% autoscale, and disabled early termination.
Deep learning methods: DiffDock sampled 40 poses using 20 inference steps with no noise on the final step and selected the top-ranked pose.ESM generated the FASTA files and embeddings used for inference.
Benchmark construction: The PoseBusters Benchmark selection produced 308 unique PDB entries containing 308 unique ligands after filtering and sequence clustering.The selection included removing ligands within 5.0 Å of protein symmetry mates.