Source-linked AI summary
ProQ3D: Improved model quality assessments using Deep Learning
Karolis Uziela, David Menéndez Hurtado, Björn Wallner, Arne Elofsson
TL;DR
Protein quality assessment seeks accurate estimates of protein-model quality from model-derived features. This work keeps the ProQ2 and ProQ3 inputs but replaces their SVM with a deep neural network, improving reported Pearson correlation to 0.90; however, top-ranked model selection does not significantly improve.
Problem
Protein quality assessment has relied on carefully tuned model features and has limited ability to capture relationships in which features provide opposite signals.
Method
The method uses the same ProQ2 and ProQ3 inputs with a deep neural network instead of a support vector machine to predict residue-level quality.
Results
Pearson correlation reaches 0.90 with ProQ3 inputs and improves local and global quality estimation relative to the corresponding SVM-based methods.
Takeaways & Limitations
Deep-learning versions improve quality estimation, but selecting better top-ranked models does not show significant improvement among the leading quality-assessment methods.
Takeaways & Limitations
Top-ranked model selection does not significantly improve, indicating that further progress will require a different approach.
Abstract
from arXiv · showhide
Summary: Protein quality assessment is a long-standing problem in bioinformatics. For more than a decade we have developed state-of-art predictors by carefully selecting and optimising inputs to a machine learning method. The correlation has increased from 0.60 in ProQ to 0.81 in ProQ2 and 0.85 in ProQ3 mainly by adding a large set of carefully tuned descriptions of a protein. Here, we show that a substantial improvement can be obtained using exactly the same inputs as in ProQ2 or ProQ3 but replacing the support vector machine by a deep neural network. This improves the Pearson correlation to 0.90 (0.85 using ProQ2 input features). Availability: ProQ3D is freely available both as a webserver and a stand-alone program at http://proq3.bioinfo.se/
1 Introduction
ProQ methods estimate protein-model quality from many tuned features, with successive versions adding profile weights and Rosetta energy terms. ProQ2 and ProQ3 use linear-kernel SVMs, while deep neural networks can model nonlinear feature relationships and substantially improve performance.
- ProQ estimates model quality rather than recognising native structures, using machine learning and many protein-model features.
- ProQ2 added profile weights, while ProQ3 added Rosetta-derived energy terms to improve predictions.
- ProQ2 and ProQ3 combine many weakly correlated features through a linear-kernel support vector machine.
- Deep neural networks identify nonlinear relationships between input features, unlike the linear combinations used by ProQ2 and ProQ3.
- Replacing the SVM with a deep neural network while retaining ProQ2 or ProQ3 inputs substantially improves performance, reaching CC=0.90 versus Pcons at 0.95.
2 Methods
The method predicts residue-level S-scores from protein-model features using models trained on CASP9 and CASP10 and evaluated on CASP11. Performance is assessed with Pearson correlations for both local and global quality.
- Features describing each protein model are used to predict single-residue quality measured by the S-score.
- Training used all CASP9 and CASP10 models, while testing used CASP11 models excluding cancelled targets and targets shorter than 50 residues.
- Pearson correlation for local and global quality was used to evaluate performance.
- The deep model used two dense hidden layers with 200 and 600 neurons and was trained with Adadelta and L2 regularization.
3 Results and Discussion
Deep-learning ProQ variants improve global and local quality estimation over the corresponding SVM-based methods on CASP11. However, the improved estimators do not significantly improve selection of top-ranked models, indicating a need for a different approach.
- Global quality is computed by summing predicted residue qualities and dividing by protein sequence length.
- Global-quality correlation improves with both ProQ2 and ProQ3 inputs for all models together and for per-target analyses.
- Figure 1 compares ProQ2 and ProQ3 with SVMs against ProQ2D and ProQ3D with Theano using CASP11 global-quality RMSE, where lower values are better.
- Local-quality prediction shows a similar improvement for correlations measured across residues and per model.
- Top-ranked model selection shows no significant improvement among the leading quality-assessment methods, including Pcons.
- The authors conclude that advancing further will require a different approach.
Supporting material for: ”ProQ3D: Improved model quality assessments
The supplied supporting-material passages identify the authors and the archived arXiv version of the work.
- The listed authors are Karolis Uziela, David Menendez Hurtado, Nanjiang Shu, Björn Wallner, and Arne Elofsson.
- The work is identified as arXiv:1610.05189v2, dated 18 October 2016.
1 Supplementary Results
On CAMEO data, deep-learning versions ProQ2D and ProQ3D outperform their SVM-based counterparts. Local prediction plots further suggest improved discrimination of low- and high-quality models in specific datasets.
- Deep-learning predictors ProQ2D and ProQ3D are superior to ProQ2 and ProQ3 on the CAMEO dataset.
- Density plots suggest deep-learning methods better distinguish low-quality CASP11 models and high-quality CAMEO models.The indicated regions are S-score < 0.1 for CASP11 and S-score > 0.9 for CAMEO.
- The benchmark compares ProQ methods with Qmean, Qprob, and the consensus method Pcons.
2 Methods
ProQ3D uses a two-hidden-layer multilayer perceptron trained with adaptive optimization and regularization methods including dropout and L2 penalties. The implementation was trained on GPUs, while prediction runs on CPUs in less than one second per model, excluding feature generation.
- The model is an MLP with hidden layers of 600 and 200 units, using ReLU nonlinearities and Adadelta optimization for mean squared error.
- Adding more hidden layers worsens results because convergence slows without added expressiveness for an already universal approximator.
- Dropout randomly removes a fraction p of hidden neurons during training; the implementation uses p = 0.5 and scales the remaining neurons by 1/p.
- L2 regularization adds a weight-dependent penalty to the loss, discouraging weights from growing unless their benefits overcome that penalty.
- Training penalties from 10^-4 to 10^-12 produced only minor effects on result quality.
- The implementation uses Keras with Theano and GPU training, but prediction runs on CPU in less than 1 second per model, excluding feature generation.