Source-linked AI summary
ProteinNet: a standardized data set for machine learning of protein structure
Mohammed AlQuraishi
TL;DR
Protein structure prediction lacks standardized resources that combine machine-learning-ready structures, high-quality MSAs, and homology-aware data splits. ProteinNet integrates these modalities, reconstructs CASP-era training and validation conditions, and supplies accessible records for model development. Across CASP 7–12, it provides six datasets with CASP-like validation difficulty and supports assessment across changing data regimes.
Problem
Existing protein structure resources do not jointly provide machine-learning-ready data, high-quality MSAs, and insulated splits that account for weakly detectable homology.
Method
ProteinNet integrates sequences, structures, MSAs, PSSMs, CASP-aligned splits, and sequence-identity thinnings in standardized machine-readable records.
Results
ProteinNet produced six CASP 7–12 datasets containing 34,557–104,059 structures, with validation sets supporting CASP-like generalization assessment.
Takeaways & Limitations
ProteinNet lowers entry costs and enables comparable training and validation of protein-structure models against CASP-era assessment conditions.
Takeaways & Limitations
CAMEO is difficult to use for apples-to-apples comparisons because its dynamic assessment requires methods to participate simultaneously, and it focuses on known folds.
Abstract
from arXiv · showhide
Rapid progress in deep learning has spurred its application to bioinformatics problems including protein structure prediction and design. In classic machine learning problems like computer vision, progress has been driven by standardized data sets that facilitate fair assessment of new methods and lower the barrier to entry for non-domain experts. While data sets of protein sequence and structure exist, they lack certain components critical for machine learning, including high-quality multiple sequence alignments and insulated training / validation splits that account for deep but only weakly detectable homology across protein space. We have created the ProteinNet series of data sets to provide a standardized mechanism for training and assessing data-driven models of protein sequence-structure relationships. ProteinNet integrates sequence, structure, and evolutionary information in programmatically accessible file formats tailored for machine learning frameworks. Multiple sequence alignments of all structurally characterized proteins were created using substantial high-performance computing resources. Standardized data splits were also generated to emulate the difficulty of past CASP (Critical Assessment of protein Structure Prediction) experiments by resetting protein sequence and structure space to the historical states that preceded six prior CASPs. Utilizing sensitive evolution-based distance metrics to segregate distantly related proteins, we have additionally created validation sets distinct from the official CASP sets that faithfully mimic their difficulty. ProteinNet thus represents a comprehensive and accessible resource for training and assessing machine-learned models of protein structure.
1 Introduction
Protein structure prediction lacks a standardized benchmark combining machine-learning-ready data, evolutionary information, and insulated splits. ProteinNet addresses this gap with integrated records and validation sets designed to reflect CASP-like generalization difficulty.
- Motivation: Standardized benchmarks can enable fair comparisons, accelerate algorithm development, and lower the barrier to entry for machine-learning researchers.
- Resource gap: Existing protein structure resources do not jointly provide machine-learning-ready structures, high-quality MSAs, and standardized training, validation, and test splits.PDB structures require postprocessing, processed sets lack evolutionary data, and existing MSA resources are unsuitable for deep homology searches.
- Existing assessment limits: CAMEO supports ongoing assessment but is difficult to use for apples-to-apples comparisons because its participating methods and targets change dynamically.
- Existing assessment limits: CAMEO focuses on proteins with known folds, making it less suitable for testing generalization to unknown regions of protein fold space.
- ProteinNet: ProteinNet provides pre-formatted sequence, structure, and MSA records together with standardized splits and validation sets that emulate CASP generalization challenges.
2 Methods
ProteinNet mirrors historical CASP information states by separating pre-CASP training data, CASP test structures, and sequence-identity-based validation sets. Its pipeline supplies evolutionary alignments, quality-controlled exemplars, machine-readable records, and lower-redundancy training thinnings.
- Data construction: The series uses CASP cutoff dates to organize data inclusion and construct non-overlapping training and validation partitions.
- Data splits: MSA-based clustering improves sensitivity for separating weakly homologous proteins before validation clusters and exemplars are selected.Sequences are redundancy-filtered, compared all-against-all with HHblits, and clustered using the resulting alignment distances.
- Data splits and thinning: Training data are reduced into 30%, 50%, 70%, 90%, 95%, and 100% sequence-identity thinnings by retaining one exemplar per sequence cluster.An 80% coverage requirement helps prevent individual domains from being grouped with multidomain proteins in training thinnings.
- Data splits: Exemplars are chosen near cluster centers and then ranked using structural quality and other criteria to preserve sequence-identity constraints while improving representative quality.
- Data access: ProteinNet records integrate sequences, secondary and tertiary structures, MSAs, and PSSMs in human-readable and binary formats with parsing code for TensorFlow.
3 Results
ProteinNet spans six CASP-era datasets and provides validation sets whose sequence-identity distances support CASP-like difficulty analyses. Earlier training snapshots make TBM targets progressively more distant, while later sequence databases yield larger alignments.
- Dataset statistics: Six ProteinNet datasets for CASP 7 through 12 contain between 34,557 and 104,059 structures.
- Dataset statistics: Training-set sizes generally increase over CASP 7–12, while sequence databases grow exponentially and later CASPs have higher-quality MSAs measured by alignment size.
- Validation suitability: ProteinNet validation sets with low sequence identity can serve as proxies for CASP FM and TBM targets, while sets above 50% identity support mutation-focused modeling.
- Difficulty across CASPs: Earlier ProteinNet training sets lose approximately 2–3% sequence-identity points per CASP, making retroactive comparisons useful for testing sensitivity to available data.This analysis was performed for TBM targets; FM targets showed no detectable homology even with up-to-date training sets.
4 Discussion
ProteinNet is presented as a community resource that lowers barriers to machine-learning research by combining relevant protein data modalities and standardized validation. Its broader utility extends beyond structure prediction to protein design, variant-effect prediction, and multitask learning applications.
- Community resource: ProteinNet lowers entry barriers by aggregating relevant data modalities in a single file format and eliminating the upfront cost of generating high-quality MSAs.Generating MSAs and PSSMs for ProteinNet 7–12 consumed over 3 million compute hours as a one-time investment.
- Community resource: Its validation sets support assessing model generalizability while training, helping make progress meaningfully ascertainable.
- Broader applications: ProteinNet can support problems beyond structure prediction because it does not prescribe which data modalities must be inputs or outputs.Examples include using structures to predict sequences for protein design or using sequence and structure to predict mutant-protein structures.
- Broader applications: Deep-learning and automatic-differentiation frameworks enable bespoke models of biological phenomena, including multitask and auxiliary-loss settings where structure can provide an additional output signal.