Source-linked AI summary
What is a meaningful representation of protein sequences?
Nicki Skafte Detlefsen, Søren Hauberg, Wouter Boomsma
TL;DR
The paper asks what makes a protein representation meaningful, given that modeling choices can produce different representations and biological interpretations. It examines transfer learning and interpretable learning, finding that several common transfer practices are suboptimal while geometry-aware representations improve interpretability. The authors conclude that no single representation is optimal for all predictive and interpretive purposes.
Problem
The paper addresses the lack of a single clear answer to what constitutes a good protein representation and why representation choice matters for extracting biological information.
Method
The authors evaluate protein representations in transfer learning and data interpretation, comparing training, aggregation, alignment, architecture, and geometry choices.
Results
The study finds that common transfer-learning practices can hurt performance, while respecting representation geometry improves robustness and supports biologically meaningful interpretation.
Takeaways & Limitations
A meaningful protein representation must be judged relative to its purpose, because representations suited to prediction may not be optimal for understanding biology.
Takeaways & Limitations
It remains open whether large transformer representations capture more information than simpler models paired with high-quality sequence alignments.
Abstract
from arXiv · showhide
How we choose to represent our data has a fundamental impact on our ability to subsequently extract information from them. Machine learning promises to automatically determine efficient representations from large unstructured datasets, such as those arising in biology. However, empirical evidence suggests that seemingly minor changes to these machine learning models yield drastically different data representations that result in different biological interpretations of data. This begs the question of what even constitutes the most meaningful representation. Here, we approach this question for representations of protein sequences, which have received considerable attention in the recent literature. We explore two key contexts in which representations naturally arise: transfer learning and interpretable learning. In the first context, we demonstrate that several contemporary practices yield suboptimal performance, and in the latter we demonstrate that taking representation geometry into account significantly improves interpretability and lets the models reveal biological information that is otherwise obscured.
Results
The paper finds that protein representations depend strongly on modeling choices and intended use: several transfer-learning practices reduce downstream utility, while geometry-aware representations improve biological interpretation.
- Representations for transfer learning: Fine-tuning reduced test performance in two of three tasks, so fixing the embedding should generally be the default under limited labeled data.The authors attribute the risk to overfitting from substantially increasing the number of free parameters.
- Representations for transfer learning: A learned Bottleneck global representation clearly outperformed averaging strategies across Stability, Fluorescence, and Remote Homology.Concatenating smaller local representations also improved performance by avoiding information loss during aggregation.
- Representations for data interpretation: Training only on β-lactamase sequences improved structure, but complete phylum separation occurred only for the Transformer and Bottleneck Resnet, while the alignment-based VAE differed substantially.Universal full-corpus models showed little or no clear phylogenetic separation for most architectures.
- Representations for data interpretation: Representation quality depends on alignment, architecture, training scope, and post-hoc dimensionality reduction, whose effects are often interdependent.The authors caution that these choices can produce substantially different representations and biological conclusions.
- Representations for data interpretation: The paper leaves open whether large transformer models can capture more information than simpler models combined with high-quality alignments.This is presented as an unresolved comparison rather than a settled limitation of any single representation.
- Representations for data interpretation: The VAE topology corresponded broadly to reconstructed phylogenetic structure, indicating that latent-space topology can preserve biologically meaningful properties.The paper therefore argues that topology should be respected when interpreting representations.
- Representations for data interpretation: Geodesic distances and paths provide more robust, biologically supported interpretation than ordinary Euclidean operations in the learned manifold.The authors motivate geodesics because straight-line interpolation can pass through uncharted regions of a star-shaped representation space.
Discussion
Meaningful protein representations must be designed for their intended purpose, because prediction and biological interpretation can require different properties. The discussion emphasizes task-aware objectives and preprocessing, biologically informed architectures and priors, and geometry-aware interpretation, while noting important scope limits.
- A single protein representation is unlikely to serve all purposes: prediction-oriented and human-interpretation-oriented representations may need different properties.The paper concludes that representation quality must always be qualified by its intended purpose.
- Designing purposeful representations: Allowing architectures to learn nonlinear relationships between local and global protein properties significantly improves performance over common linear averaging of local representations.The authors identify architecture design as a way to encode biological structure, while noting that more advanced tools may be needed.
- Designing purposeful representations: Training unsupervised models on aligned protein sequences significantly improves representation capabilities by injecting prior knowledge about comparable sequence positions.The discussion cautions that discarding established computational-biology tools may lose useful biological knowledge.
- Designing purposeful representations: Reconstruction-optimal representations are generally sub-optimal for specific transfer-learning tasks, so hyperparameters should be selected using downstream performance while guarding against overfitting.The paper anticipates combining reconstruction and task-specific objectives in semi-supervised learning.
- Analyzing representations appropriately: Geodesic distances from a Riemannian metric correlate significantly better with phylogenetic distances than ordinary Euclidean distances in representation space.The geometric analysis also suggests operational benefits for combining representations from different proteins.
- Analyzing representations appropriately: Geometric analysis was limited to variational-autoencoder latent spaces and is difficult to extend beyond single protein families because sequential language models cannot decode aggregated global representations.The authors suggest bottleneck strategies as a possible route beyond this setting.
Methods
The paper models protein sequences with variational autoencoders and defines latent-space geometry through expected distances between decoded sequences. An entropy network calibrates decoder uncertainty so geodesic calculations better reflect the data manifold.
- The VAE treats latent variables as compressed representations and generates protein sequences through pθ(X|Z).
- The entropy network maps latent distance from trainable cluster centers to uncertainty, producing maximal entropy away from observed data.
- Unlike standard VAEs, the entropy network increases decoder entropy away from data rather than allowing arbitrary confidence patterns.
- Geodesics minimize expected energy along latent curves, integrating distances between sequences decoded from points on those curves.
- Geodesic optimization discretizes the entropy landscape into a grid graph whose nodes connect to eight nearest neighbors.
Experimental details
The experiments use large-scale Pfam pre-training and multiple protein prediction datasets, supplemented by UniLanguage and β-lactamase family data for reconstruction and latent-space analyses.
- Transfer-learning experiments pre-train on 31 million Pfam protein sequences and evaluate stability, fluorescence, and remote homology across protein families.
- The reconstruction-error analysis uses UniLanguage splits designed to minimize overlap between protein families.
- Latent-space analyses use Pfam β-lactamase sequences and a DeepSequence alignment processed under different preprocessing conditions.
- Transformer and Resnet models use masked-token prediction, LSTM models use next-token prediction, and autoencoders use sequence reconstruction.
Network Architectures
The study compares TAPE-derived sequential architectures with bottleneck autoencoders and variational autoencoders under controlled training protocols. Architectures differ in sequence processing, aggregation, and latent-space design.
- The Transformer, Resnet, and LSTM architectures are parameter-matched at approximately the Transformer’s 38M parameters.
- The sequential models use attention-based aggregation, while the bottleneck Resnet autoencoder uses fixed-length padded inputs and encoder-decoder residual blocks.
- The VAE uses two 1500-unit encoder layers and decoder layers of 100 and 500 units, all with ReLU activations.
- Pre-training uses Adam, a 10^-3 learning rate with linear warm-up, 10% dropout, and dynamically selected batch sizes.
- β-lactamase models use early stopping, while VAE training uses fixed learning rates, no dropout, and batch size 16.
Additional results
Additional experiments show that reconstruction quality can diverge from downstream usefulness, dimensionality reduction can alter latent topology, and architecture, preprocessing, and initialization materially affect representations.
- Reconstruction accuracy rises with representation size, but downstream performance begins decreasing around latent sizes 500–1000.
- Reconstruction accuracy flattens after 30% of pre-training data, while all downstream tasks continue improving with additional data.
- PCA and t-SNE produce broadly robust patterns for sequential models, although t-SNE gives slightly better local phylum separation.
- For a 30-dimensional VAE latent space, PCA partly preserves the star-like structure whereas t-SNE creates a different topology.
- ESM-1b improves separation compared with smaller models and achieves phylogenetic-distance correlation comparable to family-specific models.
- Standard and variational autoencoders produce similar representations, with the VAE prior mainly setting the latent-space scale.
- Sequence reweighting improves phylum balance and preserves the reported advantage of geodesic over Euclidean distances.
- Initialization can dramatically change learned representations, unlike the usual robustness observed for model behavior and performance.