PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-level Protein Representation Learning for Blind Mutational Effect Prediction

Yang Tan, Bingxin Zhou, Yuanhong Jiang, Yu Guang Wang, Liang Hong

arXiv:2306.04899v1q-bio.QMcs.AI

TL;DR

Protein variant prediction must account for both sequence and structure, while sequence-only and structure-only approaches each have stated limitations. P13LG combines sequential and geometric representations and simulates natural selection during self-supervised training. It is evaluated across ProteinGym and two stability-focused benchmarks, and the introduction reports feasibility across this range.

  • Problem

    Sequence-only and structure-only approaches each have limitations in representing protein variant effects, despite the importance of sequence-function mapping for protein engineering.

    Source: §1 · pp. 1–2

  • Method

    P13LG combines primary-sequence and tertiary-structure representations and uses self-supervised training that simulates natural selection on perturbed amino-acid sequences.

    Source: §1 · pp. 2–3, §2.1 · p. 3

  • Results

    P13LG demonstrates feasibility across ProteinGym, covering over 80 proteins, and two single-site stability benchmarks spanning 2,967 mutants and 90 protein-condition combinations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The two stability-focused databases add assay-specific deep mutational scanning records for evaluating methods designed for protein stability.

    Source: §1 · p. 3

    Because of computational-cost constraints, P13LG does not train from scratch on an extensive corpus and requires geometric information alongside sequence data.

    Source: §6 · p. 10

Abstract

from arXiv · show

Directed evolution plays an indispensable role in protein engineering that revises existing protein sequences to attain new or enhanced functions. Accurately predicting the effects of protein variants necessitates an in-depth understanding of protein structure and function. Although large self-supervised language models have demonstrated remarkable performance in zero-shot inference using only protein sequences, these models inherently do not interpret the spatial characteristics of protein structures, which are crucial for comprehending protein folding stability and internal molecular interactions. This paper introduces a novel pre-training framework that cascades sequential and geometric analyzers for protein primary and tertiary structures. It guides mutational directions toward desired traits by simulating natural selection on wild-type proteins and evaluates the effects of variants based on their fitness to perform the function. We assess the proposed approach using a public database and two new databases for a variety of variant effect prediction tasks, which encompass a diverse set of proteins and assays from different taxa. The prediction results achieve state-of-the-art performance over other zero-shot learning methods for both single-site mutations and deep mutations.

1 Introduction

Protein variant effects can reveal how amino-acid changes alter structure and function, but mapping rules from well-studied proteins are difficult to generalize across protein families. P13LG combines sequence and structure representations for zero-shot variant prediction and is evaluated on broad and stability-focused benchmarks.

1 Introduction

  • Well-studied proteins have extensive mutational-effect records, yet protein-specific mapping rules are difficult to generalize across protein families.Examples include binding preferences, thermodynamic stability, and fluorescence.Source: §1 · p. 2
  • Sequence-only and structure-only methods each omit information: the former struggles with spatial interactions, while the latter misses long-range sequence connections.The paper motivates combining sequence and structural representations to address these respective limitations.Source: §1 · p. 2
  • P13LG combines primary- and tertiary-structure representations in a funnel-shaped pipeline, reflecting the limited availability of crystallized structures compared with protein sequences.Its stated design aims to extend the generalization and robustness of protein language models while maintaining low computational costs.Source: §1 · p. 2, Figure 1 · p. 2
  • The framework simulates natural selection during training, models joint effects in deep mutations, and requires no downstream supervision for zero-shot use.The paper connects zero-shot prediction to scarce experimental results and wet-lab cold-start settings.Source: §1 · p. 3
  • P13LG is evaluated on ProteinGym, with over 80 proteins, and two stability-focused single-site benchmarks covering 2,967 mutants across 90 protein-condition combinations.The two new databases add assay-specific deep mutational scanning records for evaluating protein-stability methods.Source: §1 · p. 3

2 Zero-shot Multi-level Protein Representation Learning

P13LG learns to recover wild-type amino-acid types from noisy sequence and local structural observations. It combines sequence-derived semantics with graph-based topology to represent proteins at multiple levels.

2 Zero-shot Multi-level Protein Representation Learning

  • The framework uses self-supervised recovery of amino-acid types from noisy local-environment observations to model selection toward well-functioning proteins.The method simulates iterative selection of well-functioning variants from harmful random mutations.Source: §2 · p. 3

2.1 Multi-level Protein Representation

  • Amino-acid observations are randomly perturbed, and the model learns a revised state less likely to be eliminated for unfavorable properties such as instability or inability to fold.The perturbation probability is tunable, and replacement frequencies are based on wild-type amino-acid frequencies in the training data.Source: §2.1 · p. 3
  • A residue graph represents local protein topology, with amino-acid nodes connected to nearby residues and geometric information encoded in node and edge attributes.The graph-based topological embedding supplements sequence relationships with spatial connections.Source: §2.1 · p. 3, §2.3 · p. 4

2.2 Semantic Encoding of Protein Sequence

  • ESM-2 encodes noisy sequences with a masked-language-modeling objective, predicting selected amino-acid identities from their sequence context.The sequence embedding is recomputed during training as amino-acid types are independently perturbed each epoch.Source: §2.2 · p. 4

2.3 Topological Encoding of Protein Structure

  • The geometric encoder uses an equivariant graph neural network to represent protein structure while preserving roto-translation equivariance and permutation invariance.The stated design avoids costly data augmentation strategies.Source: §2.3 · p. 4

3 Blind Variant Effect Prediction

P13LG is trained for blind variant-effect prediction using a frozen sequence encoder and a trainable structural graph encoder. It scores mutations against wild-type residues and evaluates predictions with rank correlation and beneficial-mutation recall.

3 Blind Variant Effect Prediction

  • P13LG is designed for blind prediction on unknown proteins, generating joint amino-acid distributions conditioned on spatial and sequential neighbors to account for epistatic effects.The model is trained self-supervised, without downstream task supervision.Source: §3 · p. 4

3.1 Model Pipeline

  • A frozen protein language model encodes sequence relationships, then EGNN layers embed the resulting residue graph for amino-acid recovery during pretraining.For a mutant, the fitness score compares predicted probabilities of mutated and wild-type amino acids at altered sites using log-odds.Source: §3.1 · pp. 4–5

3.2 Evaluation Metrics

  • Evaluation uses Spearman’s ρ for ranked prediction agreement and TPR to measure recovery of top beneficial mutations at specified cutoffs.The paper notes that absolute measurements from individual labs are typically not directly comparable across environments and experimental methods.Source: §3.2 · p. 5

4 Numerical Experiments

The experiments compare P13LG with sequence- and structure-based baselines on stability and ProteinGym mutation-effect benchmarks, then assess its design choices. P13LG performs strongly on both single-site stability prediction and deep mutations, while ablations favor equivariant graph encoding, ESM2 node features, six EGNN layers, and a 650-million-parameter ESM2 prefix.

4.1 Experimental Protocol

  • The training set contains 30,948 non-redundant CATH domains below 40% sequence identity, with proteins over 2,000 amino acids excluded.The graph encoder uses six EGNN layers, and frozen ESM2-t33 prefix embeddings provide node features.Source: §4.1 · p. 5

4.2 Variant Effect Prediction

  • P13LG consistently outperforms competitors on 100 DTm and DDG protein assays and achieves top-rank correlation with the smallest parameter count.Table 1 evaluates TPR at 5%, 25%, and 50%; Figure 2 compares parameter count with Spearman’s ρ on DTm and DDG.Source: Table 1 · p. 6, §4.2 · pp. 6–7, Figure 2 · p. 7

4.1 Experimental Protocol

  • The benchmarks cover 90 protein-condition stability assays with 2,967 single-site mutants, plus ProteinGym assays spanning diverse taxa.The stability datasets comprise 60 ΔTm assays and 30 ΔΔG assays; the ProteinGym set contains 86 DMS assays.Source: §4.1 · p. 6

4.2 Variant Effect Prediction

  • On 86 ProteinGym assays, P13LG ranks best across mutational depths and taxa, with MSA-based methods second among 27 baselines.The MSA-based methods use fewer parameters but require longer inference because they query and process additional alignment information.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Ablation Study

  • Figure 3 compares modular designs using Spearman’s correlation for single-site and arbitrary-depth mutations; roto-translation equivariance improves prediction performance.The comparison includes GCN and GAT against EGNN, with six EGNN layers and four- or six-layer alternatives for the other graph convolutions.Source: Figure 3 · p. 8, §4.3 · p. 8
  • ESM2 hidden representations outperform one-hot amino-acid node attributes in the sequence-encoding comparison.The comparison tests data-driven node attributes against hardcoded amino-acid types.Source: §4.3 · p. 8
  • Six EGNN layers suffice; deeper networks with 12 or 18 layers do not improve performance.The study identifies six layers as a moderate choice for this task.Source: §4.3 · p. 8
  • The 650-million-parameter ESM2-t33 prefix performs best, while the 3-billion-parameter t36 model degrades performance.The study finds that higher embedding dimensions and richer semantic representations do not always yield better results.Source: §4.3 · p. 8

5 Related Work

Related work spans sequence-based, structure-based, and hybrid approaches to protein representation and mutation-effect prediction. The cited work notes trade-offs in training resources and in capturing sequence context or spatial interactions.

Protein Primary Structure Embedding

  • Protein language models use Transformer-based sequence objectives, while MSA methods summarize evolutionary patterns; both can require substantial training input or resources.The passage describes autoregressive and masked-language-model objectives as well as MSA-based approaches.Source: §5 · p. 9

Protein Tertiary Structure Embedding

  • Structure-learning work encodes protein geometry for topology-sensitive tasks including binding, interface analysis, and protein-property prediction.These efforts are motivated by the role of protein structures in function and de novo design.Source: §5 · p. 9

Variant Effect Prediction

  • Supervised mutation-effect models use local-environment representations when labeled data are available; zero-shot methods instead infer fitness from evolutionary sequence alignments.Hybrid models combine MSA patterns with protein language or structure models through retrieval or attention.Source: §5 · p. 9

6 Conclusion and Discussion

P13LG cascades protein sequence and geometry representations for mutation-effect prediction, and is evaluated across protein assays and taxa. The discussion notes both potential engineering applications and limitations related to model performance and structure availability.

6 Conclusion and Discussion

  • P13LG cascades primary- and tertiary-structure embeddings in a denoising framework tailored to predicting mutational effects.It uses a prefix protein language model followed by a roto-translation equivariant graph neural network.Source: §6 · p. 9
  • The authors validate P13LG across protein function assays and taxa, including two thermal-stability databases they prepared.Source: §6 · p. 9
  • The authors flag possible use to modify harmful species and report suboptimal performance on such protein categories.Source: §6 · pp. 9–10
  • Integrated training and inference require protein geometry, so excessively long sequences without experimentally solved or computationally foldable structures may be excluded.The authors report removing an extended-length ProteinGym protein that could not be folded by computational tools.Source: §6 · p. 10

A.1 New Benchmarks: DTm and DDG

The authors construct DTm and DDG as ProThermDB-derived benchmarks for predicting single-site mutation effects on protein stability. They also evaluate 86 ProteinGym variants, excluding one protein that could not be folded by AlphaFold2.

A.1 New Benchmarks: DTm and DDG

  • DTm and DDG are new benchmarks for predicting protein stability effects of single-site mutations, sourced from ProThermDB and organized by protein–pH environment.The pairing name follows the format “PDB ID-pH level.”Source: §A.1 · p. 14
  • The benchmarks use ∆∆G and ∆Tm records, require pH-defined conditions, and limit records to single-site mutations.Figure 4 specifies ∆G and ∆∆G for DDG, and Tm and ∆Tm for DTm.Source: Figure 4 · p. 14
  • Curated assays required at least 10 records; the authors removed continuous mutations and paired sequences with AlphaFold2 structures using UniProt IDs.Source: §A.1 · pp. 14–15
  • The final benchmarks contain 60 DTm and 30 DDG protein–environment pairs.Source: §A.1 · p. 15

A.2 From TRANCEPTION: ProteinGym

  • The ProteinGym evaluation covers 86 variants from its full benchmark; one protein was excluded because AlphaFold2 failed to fold it.The full benchmark is described as roughly 1.5 million missense variants across 87 DMS assays.Source: §A.2 · p. 15

B Noise Addition Strategies

The authors perturb graph node attributes during training to emulate random mutations and compare alternative noise-addition strategies on ProteinGym using Spearman’s correlation.

B Noise Addition Strategies

  • The primary training strategy perturbs node attributes to emulate random mutations, while Table 3 compares four additional strategies using ProteinGym Spearman’s correlation.Source: B Noise Addition Strategies · p. 15, Table 3 · p. 15
  • The Mean strategy replaces a perturbed node embedding with the average embedding of other amino acids of the same type.Source: B Noise Addition Strategies · p. 15
  • Sliding Window replaces a corrupted node representation with the average of its nearest k amino-acid neighbors along the sequence.The authors motivate this strategy as retaining positional information through local neighbors.Source: B Noise Addition Strategies · p. 15
  • Gaussian Noise adds white noise with mean 0 and variance 0.5 to the node embedding vector.Source: B Noise Addition Strategies · p. 15
  • Mask replaces perturbed amino-acid tokens with <mask> or random amino-acid types, while 10% remain unchanged.Of selected tokens, 80% are masked and 10% are replaced randomly.Source: B Noise Addition Strategies · pp. 15–16

C Baseline Method Implementation Details

The authors compare baseline predictors across DTm, DDG, and ProteinGym, with model details listed in Table 4. They adjust two baseline settings to make comparisons fairer.

C Baseline Method Implementation Details

D Varient Effect Prediction on Individual Proteins

P13LG generally outperforms competitors across individual proteins and test sets, while its learning curves show a typical training–validation gap that leaves room to improve generalizability.

D Varient Effect Prediction on Individual Proteins

  • Limited DTm and DDG assay testing often yields extreme TPR values of 0 or 1 at 5%, making 25% or 50% rates more informative; 5% remains more meaningful for ProteinGym.P13LG’s overall performance is reported as surpassing baselines on all considered test sets.Source: D Varient Effect Prediction on… · pp. 16–17
  • P13LG surpasses baselines across the considered test sets and outperforms competitors for most proteins and protein-environment combinations.The individual-protein comparisons cover DTm, DDG, and ProteinGym.Source: D Varient Effect Prediction on… · pp. 16–17

E Learning Curve

  • Across nine k-neighbor and hidden-dimension combinations, early stopping supports convergence, but validation loss varies by configuration.The tested k values are 10, 20, and 30, and hidden dimensions are 512, 768, and 1280.Source: E Learning Curve · p. 17, Figure 6 · p. 17
  • A typical training–validation loss gap indicates room to improve generalizability; the authors suggest increasing encoder and training-dataset sizes.Source: E Learning Curve · p. 17

F Ablation Study

The ablation materials summarize ProteinGym correlation scores across model configurations and provide benchmark-specific correlation and TPR visualizations.

F Ablation Study

  • The ablation summary combines box plots of hyperparameter configurations with testing scores; Table 5 marks one-hot input without ESM-2 as NA and labels denoising and masked-token noise as noise and mask.Source: F Ablation Study · p. 17
  • Table 5 presents detailed ProteinGym correlation scores for the ablation study.Source: Table 5 · p. 18
  • Figures 7–9 show protein-wise Spearman correlations, while Figures 10–12 show TPR at 5%, 25%, and 50% across DTm, DDG, and ProteinGym.Source: Figure 7 · p. 19, Figure 8 · p. 20, Figure 9 · p. 21

Related papers

Loading 2306.04899v1…