PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction

Paulo Yanez Sarmiento, Pia Francesca Rissom, Manuel Pfeuffer, Marco Simnacher, Jordan F. Safer, Sumaiya Iqbal, Henrike O. Heyne, Nadja Klein, Bernhard Y. Renard

arXiv:2608.25548v1cs.LGq-bio.QM

TL;DR

Assessing protein variant effects matters, but clinical variants often remain of unknown significance and experimental measurements can be expensive. This paper extends orthogonal projection to remove biochemical-feature effects from PLM embeddings and finds that projection reduces fitness-classifier performance, while biochemical features explain part of prediction variance.

  • Problem

    Protein variant effects are important to assess, yet many clinical variants remain of unknown significance and experimental measurement is expensive.

    Source: §1 · pp. 1–2

  • Method

    The paper extends orthogonal projection to remove linear, higher-order, and interaction effects of biochemical features from PLM embeddings.

    Source: §1 · p. 2

  • Results

    Average performance fell across all four protein functions and PLMs after projection; for ESM-2 35M, average MCC fell from 0.35 to 0.15 after projection with the tabular-feature representation.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    The findings support interpreting protein-fitness embeddings through biochemical features, and the approach is transferable to a broader class of biological features.

    Source: §6 · p. 12

    The analysis is limited to human-proteome single-mutation sequences and excludes assays that do not use the UniProt reference or appear to cover only subsequences.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Recently, there has been a growing adoption of protein language models (PLMs) in biomedical science. Their embeddings provide a rich numerical representation of protein sequences which achieve state-of-the-art performance on several downstream tasks including protein fitness prediction. However, PLM embeddings are not directly interpretable and, thereby, it remains unclear what features they encode. To gain insight into which biochemical properties of the protein are driving the prediction, we leverage an orthogonal projection technique that removes linear effects of known tabular features from embeddings and extend it to high-order and interaction effects. In this way, we remove the effects of interpretable biochemical features from PLM embeddings. In an ablation study, we show that this leads to a decrease in performance for a downstream classifier trained only on the embeddings to predict protein fitness. In an additional evaluation, we find that these biochemical features explain a substantial part of the variance in the predictions of this classifier. Hence, we can show that PLM embeddings encode patterns correlated with biochemical properties and quantify their contribution to predicting protein fitness. This computationally efficient approach is not limited to the features or embeddings considered here and is readily transferable to problem settings beyond protein fitness prediction.

1 Introduction

Protein variants can alter fitness, but experimentally measuring their effects is costly, while powerful protein language model embeddings are difficult to interpret. This work extends post-hoc orthogonalization to remove biochemical-feature effects from embeddings and improve their interpretability.

Introduction

  • Protein fitness prediction estimates function relative to a reference sequence, but experimental measurement is costly and many clinical variants remain of unknown significance.The task is closely related to variant effect prediction, which frames the problem from a clinical or mutational perspective.Source: §1 · pp. 1–2
  • PLM embeddings achieve state-of-the-art performance across downstream tasks, yet their high-dimensional representations do not reveal which human-understandable features drive predictions.PLMs are trained self-supervised on large protein-sequence datasets, and their embeddings derive from 1D sequences.Source: §1 · p. 2
  • The study extends post-hoc orthogonalization beyond linear effects to remove higher-order and interaction effects of tabular features from PLM embeddings.PHO has been used in semi-structure neural networks and to remove confounding features from radiograph embeddings.Source: §1 · p. 2

2 Related Work

Related work covers methods for interpreting PLM embeddings, using PLMs for protein-fitness prediction, and applying orthogonal projection to learned representations.

Interpretation of PLMs

  • Prior interpretation work used visualizations,, attention maps for contact prediction,, neighborhood analysis, and sparse autoencoders matched to biological concepts.Source: §2 · p. 2

PLMs for Protein Fitness Prediction

  • Fitness prediction uses zero-shot substitution probabilities or supervised embedding classifiers; adds site-specific features, while task-specific fine-tuning mostly benefits embedding-input methods.Source: §2 · p. 2

Orthogonal Projection of Features

  • SSNs address identifiability when combining tabular features with deep-network latent representations, and PHO was proposed to improve SSN training and inference.[37] applied PHO to remove radiograph-embedding biases; projected metadata out during training to address confounding, while [10] used PHO for orthogonal higher- and lower-order decomposition.Source: §2 · pp. 2–3

3 Method

The method probes which fitness-relevant information PLM embeddings share with biochemical tabular features by comparing classifiers and applying orthogonal-projection ablations. It also tests conditional independence and evaluates how much of embedding-based predictions is explained by those features.

  • The workflow selects fitness-relevant tabular features, compares classifiers on features, their MLP representation, embeddings, and concatenations, then tests conditional independence.The tests ask whether embeddings add information about fitness conditional on tabular features, and vice versa.Source: §3 · p. 3
  • Projecting embeddings orthogonally to tabular features removes their linear effects; projecting to an MLP representation also targets higher-order and interaction effects.A fitted matrix D enables sample-wise removal at inference, and is calculated once by inverting a p × p matrix.Source: §3 · p. 4
  • To control for information loss from dimensionality reduction, the ablation compares actual-feature projections with projections based on shuffled features or representations.Shuffling within each fold preserves feature marginal distributions while randomizing correlations among features and with the outcome.Source: §3 · p. 4
  • Following, a separate linear evaluation model fits embedding-based prediction logits from selected features or their representation and reports adjusted R2 on the held-out test set.The hold-out set is not used for feature selection or fitting γ.Source: §3 · p. 4

4 Experiments

The experiments use filtered human-proteome ProteinGym assays and compare tabular features with PLM embeddings and their orthogonal projections using deliberately simple downstream classifiers. They assess performance with MCC, selecting regularization by AUC.

4.1 Data

  • Fitness prediction uses filtered human-proteome ProteinGym deep-mutational-scanning assays, split by function type and evaluated with stratified five-fold cross-validation.The analyzed groups contain 113k organismal-fitness, 69k activity, 51k expression, and 18k binding samples; stability is excluded because only 796 samples remain after filtering.Source: §4.1 · pp. 4–5
  • Tabular inputs capture wild-type structural properties and mutation substitutions, while embedding differences u − u0 represent changes from wild type to variant.Raw embeddings cluster by reference wild type, whereas the differences are centered in the Activity visualization; UMAP is used only for visualization.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 Models

  • An L1-regularized logistic model selects tabular features by the one-standard-error rule, and an MLP represents higher-order and interaction effects.The baseline MLP has two hidden layers of dimension 256, with a small search over model size, learning rates, and dropout.Source: §4.2 · p. 5
  • The study embeds sequences with ESM-1v and ESM-2 models, mean-pooling final-layer token embeddings and chunk-pooling sequences longer than 1,022 amino acids.ESM-2 sizes are 35M, 150M, and 650M parameters, with embedding dimensions 480, 640, and 1,280, respectively.Source: §4.2 · pp. 5–6
  • L2-regularized logistic regression predicts fitness from tabular features, embeddings, their concatenation, or projected embeddings to assess information already encoded in embeddings.The classifier is deliberately restricted in complexity so its prediction is not driven by a more expressive classification head.Source: §4.2 · p. 6

4.3 Metrics

  • Matthews correlation coefficient measures predictive performance, while AUC selects the degree of regularization.MCC ranges from −1 to 1, with 1 denoting a perfect classifier and 0 a random or constant classifier.Source: §4.3 · p. 6

5 Results

Orthogonal projection reduced protein-fitness prediction performance, indicating that PLM embeddings contain useful patterns correlated with biochemical features. Those features also explained a substantial part of the variance in embedding-based predictions and showed interpretable correlations with predicted fitness.

5.1 Performance Comparison

  • ESM-1v 650M achieved MCCs of 0.47 for Activity and 0.51 for Expression, versus 0.39 for Organismal Fitness and 0.41 for Binding.Grouping substitutions by chemical property produced similar results, suggesting less granular substitution information may suffice; L1 regularization zeroed around 30–50% of features while keeping performance within one standard error.Source: §5.1 · pp. 6–7
  • ESM-1v 650M and ESM-2 650M embeddings achieved the highest MCC for all four functions, while adding tabular features yielded small or no gains; the largest gain was for ESM-2 35M.Performance rose slightly with ESM-2 model size.Source: §5.1 · p. 6, Figure 3 · p. 7
  • Conditional-information tests found embeddings added information beyond tabular features in 15 of 16 cases, except ESM-2 35M Activity; tabular features added information beyond embeddings in only four cases.The four cases were Activity for three models and Expression for ESM-2 35M.Source: §5.1 · pp. 7–8, Table 2 · p. 8

5.2 Ablation Study

  • Orthogonal projection reduced MCC across all functions and PLMs; for ESM-1v 650M, removing linear tabular effects changed MCC by −0.09, −0.13, −0.14, and −0.10 across four functions.Projecting out tabular-feature representations reduced performance further; ESM-2 35M MCC fell from 0.35 to 0.15. Randomized projections had negligible effects, supporting the interpretation that the removed patterns were specifically correlated with biochemical features, including higher-order and interaction effects.Source: §5.2 · pp. 8–9, Figure 4 · p. 9

5.3 Evaluation Model

  • Biochemical features explained an adjusted R2 of 0.15–0.29 in ESM-1v 650M predicted logits; regular projection reduced this close to zero, while randomized projection left it essentially unchanged.Similar adjusted R2 results were observed for ESM-2 models.Source: §5.3 · pp. 9–10, Table 3 · p. 10
  • Evaluation-model coefficients linked relative solvent accessibility positively to logits across all functions and pLDDT negatively; Proline substitutions were mostly negative, as were aromatic-for-nonaromatic substitutions in three functions.Glutamine substitutions contributed mostly positively for Organismal Fitness.Source: §5.3 · pp. 10–11, Figure 5 · p. 11

6 Conclusion

The authors present an efficient projection approach for interpreting PLM embeddings by removing biochemical features. This removal substantially reduces protein-fitness prediction performance, while further evaluation finds that biochemical effects explain part of the classifier’s prediction variance.

6 Conclusion

  • Removing biochemical features substantially decreased protein-fitness classifier performance, while random-subspace controls attributed the decrease to relevant features rather than random information removal or dimensionality reduction.The evaluation also considered higher-order and interaction effects.Source: §6 · pp. 11–12
  • Biochemical features, including their higher-order and interaction effects, explain part of the variance in the embedding-based classifier’s predictions.The authors say these findings confirm previous results on PLM embeddings and predictive biochemical features for protein fitness prediction.Source: §6 · p. 12
  • The approach is not limited to the analyzed features and is readily transferable to a broader class of biological features.The authors aim to test generalization to PLMs beyond the ESM family and to PLMs fine-tuned for specific protein functions.Source: §6 · p. 12
  • The authors argue that interpretable PLMs could help elucidate disease-related biochemical mechanisms and advance drug design targeting specific proteins or functions.They motivate interpretability because PLMs can help predict the effects of variants on protein function.Source: §6 · p. 12

Disclosure of Interests

The authors report no relevant competing interests and disclose limited use of generative AI for language editing and Python code development, but not for scientific analysis or conclusions.

Disclosure of Interests

Use of Generative AI

  • The authors used ChatGPT and Claude for language editing and Python code assistance, but not for data analysis, result interpretation, or scientific conclusions.They state that they reviewed and edited all AI-generated content and take full responsibility for the manuscript.Source: Use of Generative AI · p. 12

Appendix

The appendix compares prediction performance across feature modalities and examines which tabular and structural features relate to embedding-based predictions. Real tabular-feature projections reduce performance, whereas randomized projections have negligible effects.

  • Figure 6 compares linear-classifier MCC across tabular features, their hidden-layer representation, PLM embeddings, and concatenations of embeddings with either tabular modality.Results cover four protein functions and ESM-1v 650M, ESM-2 35M, 150M, and 650M; values are fold averages with estimated standard deviations.Source: Figure 6 · p. 15
  • Across all protein functions and models, tabular-feature projection substantially reduced MCC, more so with tabular-feature representations, while randomized projections changed performance negligibly.Figure 7 reports averages across folds with estimated standard deviations.Source: Figure 7 · p. 15
  • Table 5 reports adjusted R2 for regressing ESM-2 predicted logits on tabular features or their representation before and after randomized or regular orthogonal projection.Results cover four protein functions and are averages across folds with standard deviations.Source: Table 5 · p. 16
  • Table 6 reports fold-averaged structural-feature coefficients from regressions of ESM-2 650M predicted logits, indicating significance across folds and excluding features removed during selection.Source: Table 6 · p. 16
  • Figure 8 presents fold-averaged coefficients for single and chemically grouped amino-acid substitutions fitted to ESM-2 650M predictions, with excluded features shown as zero.Red and blue indicate positive and negative contributions, respectively, to predicting the positive class.Source: Figure 8 · p. 17
Loading 2608.25548v1…