PaperlayerView exact revision on arXiv

Source-linked AI summary

PSP: Million-level Protein Sequence Dataset for Protein Structure Prediction

Sirui Liu, Jun Zhang, Haotian Chu, Min Wang, Boxin Xue, Ningxi Ni, Jialiang Yu, Yuhao Xie, Zhenyu Chen, Mengyun Chen, Yuan Liu, Piya Patra, Fan Xu, Jie Chen, Zidong Wang, Lijiang Yang, Fan Yu, Lei Chen, Yi Qin Gao

arXiv:2206.12240v1q-bio.BMcs.LG

TL;DR

Protein structure prediction research is constrained by limited accessible training data and incomplete training procedures. PSP pairs a million-level dataset with a benchmark training procedure, and the authors report that a trained model ranked first in CAMEO. They also identify sequence-database freshness as a boundary on future MSA diversity.

  • Problem

    Accessible training datasets are scarce and costly to prepare, while many structure-prediction systems provide insufficient detail about their training procedures.

    Source: §1 · p. 2

  • Method

    The paper constructs a million-level dataset combining curated experimental structures with complementary AlphaFold-predicted structures and provides a benchmark training procedure.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    The authors report that their model ranked first in CAMEO, and the full PSP resource contains 1.3M sequences with true or distilled structures.

    Source: §4.4 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that PSP could also be applied to tasks beyond protein structure prediction.

    Source: §5 · p. 9

    Most searched metagenomics databases were built before 2020, so new sequence databases are needed to increase MSA diversity and evolutionary signals.

    Source: §5 · p. 9

Abstract

from arXiv · show

Proteins are essential component of human life and their structures are important for function and mechanism analysis. Recent work has shown the potential of AI-driven methods for protein structure prediction. However, the development of new models is restricted by the lack of dataset and benchmark training procedure. To the best of our knowledge, the existing open source datasets are far less to satisfy the needs of modern protein sequence-structure related research. To solve this problem, we present the first million-level protein structure prediction dataset with high coverage and diversity, named as PSP. This dataset consists of 570k true structure sequences (10TB) and 745k complementary distillation sequences (15TB). We provide in addition the benchmark training procedure for SOTA protein structure prediction model on this dataset. We validate the utility of this dataset for training by participating CAMEO contest in which our model won the first place. We hope our PSP dataset together with the training benchmark can enable a broader community of AI/biology researchers for AI-driven protein related research.

1 Introduction

Deep-learning protein structure prediction has advanced, but access to large training datasets and detailed training procedures remains limited. PSP addresses both gaps with a million-level dataset, a benchmark procedure, and training improvements.

1 Introduction

  • Training and fine-tuning remained inaccessible to most teams because dataset access was limited and many systems lacked detailed training procedures.The paper notes that AlphaFold2 inference code was open-sourced while its training code was not, and stage-specific accuracy details were unclear.Source: §1 · p. 2
  • Only 15k sequences were available in the cited public trRosetta dataset, which the authors describe as insufficient for training at scale.Preparing million-level datasets can require millions of CPU hours, and sequence suitability and defects require expert handling.Source: §1 · p. 2
  • PSP introduces an open-source million-level protein structure prediction dataset and an associated benchmark training procedure.The paper also reports a new structure-related loss, a new sampling method for initial training, and a stabilized violation loss for fine-tuning.Source: §1 · p. 2

2 Related work

Prior work established protein-structure training datasets and deep-learning prediction methods, but the cited datasets had limits in scale or currency, while prediction approaches developed from residue-level targets to end-to-end 3D modeling.

Training dataset

  • The public trRosetta training dataset contains 15,051 sequences, while ProteinNet’s latest cutoff dates to 2016.The authors note that PDB entries with structures increased by more than 50% in the preceding five years, and characterize trRosetta as lacking the diversity needed for full-scale AlphaFold2 training.Source: §2 · p. 2

Predicting protein structure using deep learning

  • RaptorX, ESM-1b, TAPE, and ProtTrans exemplify contact-prediction approaches, while distance prediction also replaced binary contact prediction in some methods.These approaches typically predict structure-related information from residue interactions, with relatively insufficient information on interactions across long sequences.Source: §2 · pp. 2–3
  • AlphaFold2 predicts 3D atomic coordinates from aligned sequences and uses attention to represent long-range interactions, whereas RoseTTAFold uses less inference memory.For proteins with 400 residues, the paper reports 8GB for RoseTTAFold versus less than 24GB for AlphaFold2; RoseTTAFold is described as close in accuracy.Source: §2 · p. 3

3 Data Curation

PSP combines experimentally resolved structures with complementary predicted structures and associated sequence information to form a million-level training set. It also provides a non-overlapping validation set and a smaller PSP Lite option for early model adjustment.

3 Data Curation

  • The curation strategy combines experimentally resolved PDB structures with diverse sequence data and predicted structures, adding MSAs and templates to support sequence-structure prediction.The authors distinguish a true-structure dataset from a complementary distillation dataset and provide a new validation set and a PSP Lite dataset sized at one-tenth of the full training set with higher MSA quality.Source: §3 · p. 3

3.1 True structure dataset

  • The true-structure pipeline cleans PDB data into single-chain sequence-structure pairs, filters for quality, and adds MSAs and templates.The source files were split into 600k chains; the aligned sequences were searched against UniClust30 and Envdb for MSAs and against PDB70 for templates, with the final structures clustered at 40% mutual similarity into 43.9k clusters.Source: §3.1 · pp. 3–4

3.2 Distillation dataset

  • The distillation pipeline selected diverse UniRef50 sequences using sequence, length, MSA-depth, and similarity filters, then generated and filtered AlphaFold2 structures.The selection yielded 856k sequences before structure generation and 760k data bundles afterward.Source: §3.2 · p. 4

3.3 Validation set

  • The new validation set combines CAMEO targets with later PDB clusters selected to be unique and dissimilar to training data.Its labels postdate the training samples and the template and sequence-search databases, which the authors say prevents data leakage.Source: §3.3 · pp. 4–5

3.4 Dataset summary

  • The combined resource contains 570k experimental-structure sequences and 760k predicted-structure sequences, organized for integration into a training set of over a million samples.The new validation set contains 513 samples described as free of data leakage.Source: §3.4 · p. 5

4 Benchmark

The benchmark applies a two-stage training procedure to AlphaFold2 on PSP, compares training against trRosetta, and evaluates models on CASP14, a new validation set, and CAMEO. The reported results include higher CASP14 TM-score with PSP than trRosetta and top CAMEO rankings for the authors’ models.

4.1 Experiment setup

  • The experiments trained AlphaFold2, monitored TM-score on 87 CASP14 sequences, and excluded sequences released after May 13, 2020 to prevent data leakage.Training used batch size 1 on 128 Ascend 910 accelerators; 75k steps corresponded to 9.6 million samples.Source: §4.1 · p. 5

4.2 Datasets comparison

  • 0.8 TM-score on CASP14 was reached with PSP, versus 0.7 with trRosetta, after both trainings used 1.5 million samples.TrRosetta’s score rose faster early, while PSP training used 1.5 epochs compared with 100 epochs for trRosetta.Source: §4.2 · p. 5

4.3 Training Performance

  • Training followed an initial-training and fine-tuning process, with modifications to both stages.Source: §4.3 · p. 5

4.3.1 Initial training

  • Without dropout, initial training reached 0.75 in 10k steps and 0.84 by 45k, but accuracy later fell below 0.8.With dropout, it took 20k steps to reach 0.75 and converged to 0.79.Source: §4.3.1 · p. 6
  • The new sampling scheme includes all distillation data each epoch and samples true-structure data with replacement, reaching a final accuracy of 0.86.It makes all distillation samples pass through training about every 10k steps; this result was equivalent to the released DeepMind checkpoint trained with two additional fine-tuning stages.Source: §4.3.1 · p. 6
  • Adding C and N atom-position errors to the auxiliary loss adds residue-orientation information and reached 0.86 without a later accuracy decrease.The baseline auxiliary FAPE term considered only Cα positions, which did not regularize residue orientation.Source: §4.3.1 · pp. 6–7

4.3.2 Fine-tuning

  • Clipping the clash term at 5.0 stabilized fine-tuning and maintained 0.86 TM-score on CASP14, while violations approached zero after thousands of steps.Fine-tuning used a 1e-4 learning rate and crop length 384; dropout was turned off after TM-score dropped rapidly with dropout. On the new validation set, lDDT was 0.80, matching the DeepMind checkpoint, and same-setting comparisons were at least equivalent to the original model.Source: §4.3.2 · pp. 7–8

4.4 Cameo Assessment

  • First place in the March 25–April 16, 2022 CAMEO evaluation went to model 1.1; the scoring model ranked second and model 1.2 fourth.Model 1.1 used templates, while model 1.2 did not.Source: §4.4 · p. 8

5 Limitations and Future Works

PSP prioritizes sequence diversity and aims to expand beyond PDB data. Its distillation set could grow by loosening diversity constraints, while greater MSA diversity and evolutionary signals require newer sequence databases.

5 Limitations and Future Works

  • Most searched metagenomics databases predate 2020, so newer sequence databases are needed to increase MSA diversity and add evolutionary signals.The authors also note that the distillation set could expand by loosening diversity constraints as sequence databases naturally grow.Source: §5 · p. 9
  • PSP prioritizes sequence diversity and aims to expand beyond PDB data.The authors say the dataset’s applications could also extend to sequence/MSA pre-training, protein design, and sequence or structure generation.Source: §5 · p. 9
  • The authors demonstrate PSP’s effectiveness by training a state-of-the-art protein structure prediction model from scratch.Source: §5 · p. 9

6 Conclusion

The paper presents a 1.3-million-sequence protein structure prediction dataset and a training benchmark, with model improvements and a first-place CAMEO result. The authors provide training details as a benchmark for researchers and make the work publicly available.

6 Conclusion

  • The model trained using PSP won first place in the CAMEO contest.Source: §6 · p. 9
  • PSP contains 1.3M sequences with true or distilled structures and is described as having greater coverage, diversity, and depth than existing open datasets.Source: §6 · p. 9
  • The authors provide a benchmark for training state-of-the-art protein structure prediction models, including details of the training procedure.Source: §6 · p. 9
  • The authors report making all their work, including the dataset, open to the public.Source: §6 · p. 9

Data and Code Availability

The training code is available through MindSPONGE, and the full dataset is intended for public hosting at Peking University/Changping Laboratory. The dataset and code have stated licenses.

Data and Code Availability

A.1.1 Database in use

The database sources used in data curation are listed in Table 4, and the authors state that both those sources and PSP use a Creative Commons 4.0 license.

A.1.1 Database in use

  • Table 4 lists the databases used in data curation, and the authors state that all are available under Creative Commons 4.0.Source: §A.1.1 · p. 10
  • The PSP dataset is under the same Creative Commons 4.0 license as the listed databases.Source: §A.1.1 · p. 10

A.1.2 True structure dataset cleaning

The true-structure cleaning pipeline standardizes downloaded PDB chains and aligns their residue numbering and sequence content with processed FASTA files. Missing structural residues are retained as masked UNK positions to keep sequence and structure continuous.

A.1.2 True structure dataset cleaning

  • PDB structures were fetched as single chains and matched to the corresponding FASTA files using author chain names.The FASTA files were downloaded from the same PDB list and split into single-chain files.Source: §A.1.2 · p. 10
  • Cleaning removes alternative conformations, converts specified phosphorylated residues and selenomethionine to standard residues, and filters other heteroatoms.For arginine, the NH1/CD and NH2/CD distances are checked to select the appropriate atom lines.Source: §A.1.2 · p. 10
  • Cleaned structures are converted to sequences and aligned to raw FASTA files so sequences can be trimmed to match the cleaned structures.Residues are renumbered from 1, and gaps are filled with UNK so sequence and structure remain continuous while missing coordinates can be masked.Source: §A.1.2 · p. 10

A.1.3 Compensatory filtering in distillation set

The merged sections describe template-based filtering and processing of the dataset, plus evaluation of training and CAMEO models. The dataset is also reduced into a smaller PSP lite version and organized for access and use.

A.1.3 Compensatory filtering in distillation set

  • A normalized HHsearch template score below 0.4 was used to select distillation sequences dissimilar to solved structures, leaving about 800k of 4M candidates.The score is sum_probs divided by query length; the figure compares score distributions across four query-length intervals.Source: §A.1.3 · p. 10, Figure 6 · p. 11

MSA search with MMseqs2

Template search with HHsearch

  • HHsearch template searches were run after the full protocol for the true-structure set but before large-scale MSA searching for the distillation set.The protocol used default parameters except maxseq was set to 1,000,000.Source: Template search with HHsearch · pp. 11–12

Structure prediction with AlphaFold

  • Distillation structures were predicted before confidence filtering with AlphaFold v2.0.1 using MMseqs2 MSAs and a fake template.The prediction used model_1_ptm and the specified July 14, 2021 parameter set.Source: Structure prediction with AlphaFold · p. 12

A.1.5 PSP lite

  • PSP lite reduces raw database size by 80%; after MSA filtering and depth limits, it contains 440k true-structure and 640k distillation samples.Samples with fewer than 128 MSA sequences were removed; the resulting datasets are 185GB and 167GB, respectively.Source: §A.1.5 · p. 12

A.3.1 Two-stage training evaluation

  • Fine-tuning improved both all-atom lDDT accuracy and C-alpha TM-scores relative to initial training on the new validation set.Figure 8 presents per-sample comparisons between the two stages and comparisons with standard AlphaFold.Source: §A.3.1 · p. 12, Figure 8 · p. 14

A.2 Statistics of Dataset

  • Dataset statistics are summarized separately for true-structure and distillation data, with per-sequence JSON metadata for each set.Metadata includes sequence length, MSA depth, and template count; true-structure records also include release date, resolution, and experimental method, while distillation records include predicted confidence.Source: §A.2 · p. 12, Figure 7 · p. 13

A.3.2 Details for CAMEO contest

  • Model 1.1 ranked first, the scoring model second, and model 1.2 fourth in the CAMEO cycle; the models remained best on the 51-target common-submission set.The comparison covered March 25 to April 16, 2022.Source: §A.3.2 · pp. 13–14, Table 2 · p. 15, Table 3 · p. 16
Loading 2206.12240v1…