Evaluating Protein Transfer Learning with TAPE

Roshan Rao; Nicholas Bhattacharya; Neil Thomas; Yan Duan; Xi Chen; John Canny; Pieter Abbeel; Yun S. Song

Source-linked AI summary

Evaluating Protein Transfer Learning with TAPE

Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, Yun S. Song

arXiv:1906.08230v1Jun 19, 2019cs.LGq-bio.BMstat.ML

TL;DR

Protein representation learning lacks standardized, biologically relevant evaluation despite rapidly growing unlabeled sequence databases and costly labels. TAPE addresses this gap with five tasks, curated generalization-focused splits, and broad model comparisons. Self-supervised pretraining helps broadly, but learned features still trail state-of-the-art alignment-based features on several tasks, leaving substantial room for improvement.

Problem
Protein sequence databases are growing much faster than annotated subsets, while existing semi-supervised learning evaluations use fragmented datasets and inconsistent splits.
Method
TAPE benchmarks protein representation-learning approaches on five biologically relevant tasks using curated splits designed to test biologically relevant generalization.
Results
Self-supervised pretraining improves performance for almost all models on all tasks, but alignment-based features outperform learned features on several structure-related evaluations.
Takeaways & Limitations
TAPE provides a common benchmark for protein modeling, while the results identify architecture and modeling improvements as important opportunities.
Takeaways & Limitations
No single self-supervised model performs best across all protein tasks, and language modeling alone may not suffice to surpass state-of-the-art performance.

Abstract

from arXiv · show

Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, and test splits to ensure that each task tests biologically relevant generalization that transfers to real-life scenarios. We benchmark a range of approaches to semi-supervised protein representation learning, which span recent work as well as canonical sequence learning techniques. We find that self-supervised pretraining is helpful for almost all models on all tasks, more than doubling performance in some cases. Despite this increase, in several cases features learned by self-supervised pretraining still lag behind features extracted by state-of-the-art non-neural techniques. This gap in performance suggests a huge opportunity for innovative architecture design and improved modeling paradigms that better capture the signal in biological sequences. TAPE will help the machine learning community focus effort on scientifically relevant problems. Toward this end, all data and code used to run these experiments are available at https://github.com/songlab-cal/tape.

1 Introduction

TAPE addresses fragmented evaluation of semi-supervised protein learning by benchmarking diverse biologically relevant tasks and models. Self-supervised pretraining generally helps, but alignment-based features remain stronger on some tasks.

Protein databases are expanding exponentially while experimentally annotated subsets lag because meaningful labels require substantial resources and expertise.
TAPE introduces five biologically relevant supervised tasks spanning diverse aspects of protein understanding to systematically evaluate learned protein embeddings.
The benchmark uses biologically relevant splits and compares recurrent, convolutional, attention-based, and recently proposed semi-supervised models.
Self-supervised pretraining improves performance for almost all models on all downstream tasks, while architecture performance varies significantly across tasks.
Alignment-based features outperform self-supervised features on secondary structure and contact prediction, whereas learned features perform better on remote homology detection.
Self-supervision is promising for protein modeling, but substantial improvements are still needed before self-supervised models achieve breakthrough performance.

2 Background

Protein modeling represents proteins as amino-acid sequences while also targeting their structure and evolutionary relationships. Semi-supervised learning seeks to exploit abundant unlabeled sequences alongside scarce labeled data, and alignment methods provide a central comparison point.

Proteins are represented as sequences of discrete amino-acid characters from a fixed 25-character alphabet.
Protein structure spans primary, secondary, and tertiary levels, with folding from sequence into three-dimensional structure as a fundamental biochemical problem.
Homologs are proteins with different sequences that nevertheless share evolutionary or functional relationships.
Sequence identity measures exact amino-acid matches between aligned subsequences and helps prevent information leakage between data splits.
Alignment-based methods compare test proteins with databases using scoring systems, probabilistic models, or both, then use the resulting information directly or in downstream models.
Semi-supervised learning jointly leverages unlabeled and labeled data to maximize performance on a supervised task.

3 Related Work

Earlier protein benchmarks and semi-supervised methods provided important foundations but differed in scalability, tasks, datasets, and split strategies. These inconsistencies make direct comparison of representation-learning approaches difficult.

CASP is a major protein-modeling benchmark focused on structure prediction, while ProteinNet supplies curated splits and contributes TAPE’s contact-prediction task.
ProteinNet applies sequence-identity filtering, and TAPE broadens evaluation beyond structure prediction with additional protein-biology tasks.
Kernel-based protein pretraining improved network prediction and homolog detection but did not scale beyond hundreds of thousands of unlabeled examples.
Prior representation-learning studies used different datasets and data-splitting strategies across structure, contact, and protein-engineering tasks.
Such varied task evaluation makes relative merits of self-supervised modeling approaches difficult to assess and hinders efficient progress.

4 Datasets

TAPE curates five biologically relevant protein tasks with task-specific splits designed to test generalization across evolutionary distances and local sequence neighborhoods. The tasks cover structure prediction, evolutionary understanding, and protein engineering, with explicit definitions, impacts, generalization targets, and metrics.

Task splits are designed to test biologically relevant generalization across large evolutionary distances and real-life scenarios.The benchmark provides specific training, validation, and test splits for each dataset.
TAPE curates five downstream prediction tasks spanning structure prediction, evolutionary understanding, and protein engineering.The datasets contain between 8 thousand and 50 thousand training examples.
Structure prediction: Secondary structure prediction maps each amino acid to Helix, Strand, or Other labels and reports per-amino-acid accuracy on CB513.The task tests local-structure learning under splits filtered at 25% sequence identity.
Structure prediction: Contact prediction maps amino-acid pairs to contact labels for pairs less than 8Å apart and reports precision among the L/5 most likely medium- and long-range contacts.The task uses ProteinNet CASP12 data and tests understanding of global protein context.
Evolutionary understanding: Remote homology detection classifies proteins into 1,195 fold labels while holding out entire evolutionary groups and reporting fold-level accuracy.This split tests structural-similarity detection across large evolutionary gaps.
Protein engineering: Fluorescence prediction regresses log-fluorescence, training on a Hamming distance-3 GFP neighborhood and testing on variants with four or more mutations.Performance is measured using Spearman’s ρ to assess ranking of unseen mutation combinations.
Protein engineering: Stability prediction measures the extreme concentration at which a protein maintains its fold, training broadly and testing on Hamming distance-1 neighbors of top candidates.The task reports Spearman’s ρ and evaluates local refinement from broad sequence sampling.

5 Models and Experimental Setup

The benchmark compares self-supervised and protein-specific pretraining strategies across recurrent, convolutional, attention-based, and previously proposed architectures. It uses standardized downstream evaluation and alignment-based and one-hot baselines to compare representations systematically.

Pretraining objectives: The study evaluates next-token and masked-token prediction, including bidirectional context for next-token modeling.It also examines supervised pretraining on contact prediction and remote homology detection.
Architectures: Three implemented architectures are an LSTM, Transformer, and dilated residual network, with approximately matched parameter counts.The Transformer has 12 layers, 512 hidden units, 8 attention heads, and 38M parameters.
Architectures: Two previously proposed models are also benchmarked: Bepler et al.’s bidirectional language model and Alley et al.’s unidirectional mLSTM.These architectures differ substantially from the newly implemented models.
Training: The Transformer and ResNet use masked-token prediction, whereas the LSTM and both prior models use next-token prediction.All self-supervised models are trained on four NVIDIA V100 GPUs for one week.
Baselines: Learned representations are compared with one-hot sequence encodings and alignment- or HMM-based features.Alignment-based inputs are omitted for protein engineering because they return identical features for proteins differing by one amino acid.

6 Results

Self-supervised pretraining improves performance across nearly all evaluated models and tasks, but its benefits vary by task and remain below alignment-based methods in some settings.

Language modeling: Heldout-family accuracy is consistently lower than random-split accuracy, indicating weaker out-of-distribution generalization.The language-modeling results show this pattern across the evaluated architectures.
Downstream tasks: Self-supervised pretraining improves overall performance across almost all models and downstream tasks.
Fluorescence: Fluorescence predictions cluster bright and dark proteins but still misclassify many proteins.The fluorescence distribution is bimodal, with bright and dark modes.
Protein engineering: 70% accuracy versus 68% accuracy distinguishes beneficial from deleterious mutations for the best pretrained and non-pretrained stability models, respectively.Beneficial mutations are defined by predicted stability exceeding the parent protein’s predicted stability.
Contact prediction: Pretraining captures more long-range contact information, but alignment features produce a sharper map with higher specificity for fewer well-spaced contacts.The longest-range contact shown spans positions 8 and 136.

7 Discussion

Alignment-based inputs outperform current self-supervised models on several tasks, while model rankings differ substantially across tasks, motivating multi-task evaluation.

Comparison to state of the art: Alignment-based inputs outperform current self-supervised models on secondary structure, contact prediction, and remote homology classification.NetSurfP-2.0 reaches 85% accuracy versus 75% for the best model; RaptorX reaches 0.69 precision versus 0.49; DeepSF reaches 41% accuracy versus 26%.
Need for multiple benchmark tasks: The Transformer performs worst on secondary structure but best on fluorescence and stability, whereas the ResNet performs far worse on fluorescence.The ResNet’s fluorescence Spearman’s ρ is 0.21 compared with 0.67 for the LSTM.
Need for multiple benchmark tasks: Performance on a single task does not capture the full extent of a trained model’s knowledge and biases, supporting multi-task benchmarks such as TAPE.

8 Future Work

The authors identify substantial room for improving protein representation learning as labeled data remain scarce and no single self-supervised model performs best across all tasks.

Research direction: The growing gap between labeled and unlabeled protein data means self-supervised learning will remain important for computational protein modeling.
Research direction: No single self-supervised model performs best across all protein tasks.
Future work: Future work should explore architectures, training procedures, unsupervised tasks, and the relationship between alignment-based and learned representations.The authors question whether language modeling alone is sufficient and suggest protein-specific tasks may be necessary to exceed the state of the art.
Benchmarking: TAPE provides a systematic evaluation framework intended to support broader machine-learning participation in protein representation learning.

A.1 Dataset Details

The dataset appendix describes how TAPE’s supervised tasks are documented, including dataset sources and splits, labeling procedures, and dataset sizes.

Dataset documentation: Table S1 reports the sizes of all training, validation, and test sets.
Dataset documentation: The appendix provides details on dataset sources, preprocessing decisions, data splitting, and challenges in obtaining supervised labels.
Dataset documentation: Each task description identifies the dataset source and creation of train/test splits, along with the current approach to acquiring supervised labels.

A.1.1 Secondary Structure Details

Secondary-structure evaluation uses identity-filtered datasets and multiple held-out test sets to assess generalization to proteins unlike those seen during training.

The evaluation uses CB513, CASP12, and TS115 as three test sets.
The filtering tests whether models generalize when test proteins are not closely related to training proteins.
Experimental secondary-structure labeling requires high-resolution imaging, which can take weeks to years and cost over $200,000.The passage names Cryo Electron-Microscopy and X-Ray Crystallography as examples.

A.1.2 Contact Prediction Details

Contact prediction and related benchmark tasks test generalization across evolutionary distance, mutation neighborhoods, and protein topologies using fixed supervised architectures and multiple evaluation metrics.

Contact Prediction: Contact-prediction data use ProteinNet with training and validation filtered at 30% sequence identity and a CASP12-based test set.This tests generalization to proteins not closely related to training proteins.
Contact Prediction: Contact labeling requires a protein’s full 3D structure, which in turn requires imaging.
Remote Homology: The remote-homology benchmark holds out entire superfamilies, testing generalization across evolutionary distance while preserving structure.Fold-level accuracy is reported on the held-out set.
Remote Homology: The remote-homology task matters for finding new superfamilies with the same fold, including in metagenomic settings.
Protein Function: Deep Mutational Scanning splits training and validation within Hamming distance 3, while testing uses proteins at distances 4–15.DMS becomes vanishingly sparse beyond Hamming distance 2.
Protein Stability: Stability data use four experimental rounds for training and validation, with testing on seventeen one-mutation neighborhoods around promising proteins.The selection process could be improved by better computational models, creating a potential virtuous cycle for labeled data.
Experimental Setup: The benchmark fixes one supervised architecture per task and compares one-hot, alignment-based, and neural-network features without hyperparameter tuning or major architecture optimization.Remote homology additionally uses an attention-weighted mean to produce a sequence-length-invariant protein embedding.
Secondary Structure: Secondary-structure evaluation covers three- and eight-class DSSP classification, with alignment features outperforming alternatives by a large margin.