Learning protein sequence embeddings using information from structure

Tristan Bepler; Bonnie Berger

Source-linked AI summary

Learning protein sequence embeddings using information from structure

Tristan Bepler, Bonnie Berger

arXiv:1902.08661v2Feb 22, 2019cs.LGq-bio.BMstat.ML

TL;DR

Protein structural information is scarce, and sequence similarity alone often misses related folds after sequence divergence. The paper learns contextual sequence embeddings with soft symmetric alignment and contact-based multitask supervision, outperforming sequence methods and TMalign for structural-similarity prediction while improving transmembrane prediction.

Problem
Protein structures are unavailable for most known sequences, while sequence-based similarity methods struggle to recognize related structures when sequences have diverged.
Method
A biLSTM maps sequences to position-specific embeddings trained with global SCOP similarity, soft symmetric alignment, ordinal regression, and residue-contact supervision.
Results
The multitask model outperforms sequence-based methods and TMalign for structural-similarity prediction and improves state-of-the-art transmembrane prediction.
Takeaways & Limitations
Structure-informed sequence embeddings transfer to other protein sequence problems while capturing both global similarity and local structural information.

Abstract

from arXiv · show

Inferring the structural properties of a protein from its amino acid sequence is a challenging yet important problem in biology. Structures are not known for the vast majority of protein sequences, but structure is critical for understanding function. Existing approaches for detecting structural similarity between proteins from sequence are unable to recognize and exploit structural patterns when sequences have diverged too far, limiting our ability to transfer knowledge between structurally related proteins. We newly approach this problem through the lens of representation learning. We introduce a framework that maps any protein sequence to a sequence of vector embeddings --- one per amino acid position --- that encode structural information. We train bidirectional long short-term memory (LSTM) models on protein sequences with a two-part feedback mechanism that incorporates information from (i) global structural similarity between proteins and (ii) pairwise residue contact maps for individual proteins. To enable learning from structural similarity information, we define a novel similarity measure between arbitrary-length sequences of vector embeddings based on a soft symmetric alignment (SSA) between them. Our method is able to learn useful position-specific embeddings despite lacking direct observations of position-level correspondence between sequences. We show empirically that our multi-task framework outperforms other sequence-based methods and even a top-performing structure-based alignment method when predicting structural similarity, our goal. Finally, we demonstrate that our learned embeddings can be transferred to other protein sequence problems, improving the state-of-the-art in transmembrane domain prediction.

1 INTRODUCTION

The paper learns protein sequence embeddings that capture structural information despite weak sequence–structure correspondence. A multitask framework uses global structural similarity and residue contacts, outperforming sequence-based methods and TMalign for structural-similarity prediction while transferring to transmembrane prediction.

Protein structure is essential for understanding function, but experimental structures cover only a tiny fraction of known proteins.
Divergent sequences can share structural folds, limiting sequence-based transfer of knowledge between structurally related proteins.
The framework learns position-specific biLSTM embeddings using weak supervision from global structural similarity and within-protein residue contacts.
The model dramatically outperforms sequence-based methods and even TMalign when predicting shared SCOP structural similarity using sequence alone.
SSA outperforms alternative alignment methods, while contact-prediction training further improves structural-similarity and secondary-structure prediction.
The learned embeddings improve state-of-the-art transmembrane prediction, supporting transfer to other protein machine-learning problems.

2 RELATED WORK

Prior protein sequence embeddings mainly use fixed-size k-mer or alignment-derived representations, which do not support sequence labeling and often depend on high sequence similarity. The paper instead uses contextual, position-specific embeddings with a symmetric soft alignment designed for efficient learning.

Earlier protein embedding methods primarily learn fixed-size representations from k-mer co-occurrence or alignment scores.
Fixed-size representations cannot be directly used for sequence-labeling tasks such as active-site or transmembrane prediction.
Contextual bidirectional recurrent embeddings motivate representations that encode the structural context of each amino acid and transfer across tasks.
Word-alignment and document-similarity methods address unaligned sequence elements, but are not designed for contextual neural embedding learning and may be computationally expensive.
SSA is symmetric, memoryless, interpretable, and memory-efficient, scaling with the product of sequence lengths.
Protein fold-recognition methods based on sequence homology, alignments, or profile HMMs are mainly appropriate for proteins with high sequence similarity.

3 METHODS

The framework encodes amino acid sequences into contextual vectors, compares proteins through soft symmetric alignment and ordinal SCOP regression, and adds residue-contact prediction as a second training task. These components are trained end-to-end, optionally using frozen pretrained language-model states.

The framework combines a multilayer biLSTM encoder, SSA with ordinal regression for global structural similarity, and a convolutional contact-prediction module.
The biLSTM maps each protein sequence to same-length contextual vectors whose positions incorporate information from surrounding amino acids.
A pretrained bidirectional LSTM language model supplies hidden states alongside one-hot amino-acid inputs, while its parameters remain frozen during supervised training.
Structural similarity is classified by the most specific shared SCOP level, represented as an ordered five-class label from no similarity through family similarity.
SSA compares arbitrary-length embedding sequences by inferring soft symmetric alignment and calculating similarity from aligned vector representations.
Ordinal regression predicts whether similarity reaches each SCOP level, with monotonic probabilities enforced by nonnegative slope parameters.
Contact prediction provides position-level supervision by classifying residue pairs whose Cα atoms lie within 8Å in the protein structure.
Pairwise contact features concatenate absolute embedding differences and element-wise products, then convolutions produce an N×N contact-probability matrix.

4 RESULTS

The full SSA embedding model is evaluated for structural similarity, ablations, and transfer to secondary-structure and transmembrane prediction. It outperforms comparison methods and benefits from SSA, contact prediction, pretrained language-model inputs, and a deeper encoder.

Structural similarity prediction: On ASTRAL 2.06 and 2.07, the full SSA model outperforms all comparison methods across structural-similarity metrics.Comparisons include NW-align, phmmer, HHalign, and TMalign; metrics include accuracy, Pearson’s and Spearman’s correlations, and average precision across SCOP levels.
Structural similarity prediction: 0.95 versus 0.79 accuracy, 0.91 versus 0.37 Pearson correlation, and 0.69 versus 0.23 Spearman correlation are achieved over HHalign on ASTRAL 2.06.The comparison is against the next-best sequence comparison method and requires no database search for sequence profiles.
Ablation study of framework components: SSA outperforms uniform alignment and mean-embedding comparisons across the combined structural-similarity and local-structure evaluation.Mean embeddings approach SSA on secondary-structure prediction but perform significantly worse for SCOP similarity; uniform alignment shows the opposite pattern.
Ablation study of framework components: Secondary-structure accuracy is 0.487 with SSA embeddings versus 0.444 with 3-mer features, and rises to 0.630 when contact prediction is included.The contact-prediction task improves both structural-similarity prediction and secondary-structure prediction.
Ablation study of framework components: Including pretrained language-model hidden states raises SCOP similarity accuracy from 0.898 to 0.938, while simpler encoders underperform the 3-layer biLSTM.The language-model hidden states alone are not sufficient for high performance on the structural-similarity task.
Transmembrane prediction: Transmembrane predictions rank first or tied for first in three of four categories and achieve 0.89 accuracy versus 0.87 for TOPCONS.The improvement comes from replacing the CRF potential function with a function of the structural embedding vectors while retaining TOPCONS’s hidden-state grammar.

5 CONCLUSION

The paper introduces SSA-based contextual protein embeddings learned from global structural similarity and reports strong structural and transmembrane prediction performance, while identifying a single-domain scope limitation.

SSA learns contextual sequence embeddings with weak supervision from a global similarity measure.
The model outperforms competing methods, including TMalign, when predicting protein structural similarity.
Training on SCOP focuses exclusively on single-domain protein sequences, so embeddings may differ for the same domain in multi-domain contexts.
The resulting embeddings improve state-of-the-art transmembrane region prediction and can be applied to other protein prediction tasks.

A APPENDIX

The appendix documents the embedding architecture, SCOP hierarchy, dataset sequence-length statistics, and encoder architecture comparisons.

Embedding model: Figure 2 illustrates an embedding model that combines bidirectional language-model states and amino-acid one-hot inputs before a bidirectional LSTM encoder.
SCOP hierarchy: Figure 3 illustrates the hierarchical organization of protein domains in SCOP.
Dataset statistics: Table 4 reports mean, standard deviation, minimum, and maximum sequence lengths for the SCOPe datasets.
Encoder comparisons: Table 5 compares encoder architectures on the ASTRAL 2.06 test set using language-model inputs and SSA without contact prediction.

A.1 LANGUAGE MODEL TRAINING

The pretrained bidirectional language model predicts each amino acid from both preceding and following sequence context using Pfam protein domains.

The language model was trained on 21,827,419 Pfam protein domain sequences.
The forward and reverse probabilities are combined by summing their log probabilities.
The model uses a shared-parameter 2-layer LSTM with 1024 units per layer and a linear transformation to 20 amino-acid outputs.

A.2 HYPERPARAMETER SELECTION

Hyperparameters were selected on a held-out SCOP validation set, where amino-acid resampling and contact-prediction multitask weighting improved structural-similarity accuracy.

Validation procedure: The validation set contains 2,240 held-out training sequences and 100,000 randomly sampled sequence pairs.
Resampling probability: Amino-acid resampling with probability 0.05 slightly improves structural-similarity results with or without language-model inputs.
Multitask loss weight: All models trained with contact prediction outperform models without contact prediction, with accuracy increasing as λ decreases.
Multitask loss weight: λ = 0.1 was selected because it achieved the best validation-set prediction accuracy among the evaluated values.

A.3 TIME AND MEMORY REQUIREMENTS

The model’s embedding computation scales linearly with sequence length and is fast enough for long proteins on a single GPU, while SSA comparison scales with the product of sequence lengths.

Embedding time and memory scale linearly with sequence length.
0.03 ms per amino acid embedding time was measured on an NVIDIA V100.
Less than 9 GB of RAM sufficed to embed 130,000 amino acids on a single GPU.
SSA computation scales as O(nm) for sequences of lengths n and m and is easily parallelized on a GPU.
Computing SSA for 237,016 sequence pairs required 101 seconds total, or 0.43 ms per pair on average.

A.4 STRUCTURAL SIMILARITY PREDICTION BENCHMARKS

The structural-similarity benchmark compares sequence-alignment baselines under specified scoring procedures and calibrates classification thresholds using sampled training pairs.

NW-align used BLOSUM62 with gap-open and gap-extension penalties of -11 and -1, respectively.
phmmer scored each sequence pair in both directions with the --max option and averaged the two scores.
HHalign built each sequence’s multiple alignment with up to two HHblits iterations against uniclust30 before averaging bidirectional HMM-HMM scores.
TMalign compared aligned protein structures rather than sequence-only inputs.
Classification thresholds were chosen to maximize accuracy using 100,000 sequence pairs sampled from the ASTRAL 2.06 training set.

A.5 CONTACT PREDICTION PERFORMANCE

The contact-prediction evaluation reports performance across all and distant contacts on ASTRAL and CASP12 datasets, with comparisons to co-evolution-based methods on CASP12.

The full SSA model was evaluated on all contacts (|i −j| ≥2) and distant contacts (|i −j| ≥12) using precision, recall, F1, AUPR, and top-L precision.
The ASTRAL evaluation covered the 2.06 test set and the 2.07 new test set.
CASP12 evaluation used 21 free-modelling protein domains and compared SSA with co-evolution-based contact predictors.
The model dramatically outperformed comparison methods for predicting all contacts but performed worse for predicting only distant contacts.
The distant-contact disadvantage is attributed to training on all contacts, whereas co-evolution methods target distant contacts.
The SSA model nevertheless outperformed GREMLIN on distant-contact AUPR, and its embeddings may complement co-evolution features in dedicated predictors.