Source-linked AI summary
Learning protein constitutive motifs from sequence data
Jérôme Tubiana, Simona Cocco, Rémi Monasson
TL;DR
WW Type II and III ligand classes remain difficult to distinguish because domains often bind both. The paper applies RBMs to protein sequences, recovering interpretable structural and functional patterns and producing diverse lattice sequences with high fitness.
Problem
WW Type II and III ligand classes are difficult to distinguish because domains often have high affinity for both ligand types.
Source: Inferred weights and interpretation · p. 9Method
RBMs model protein sequences jointly with hidden representations, using sparse weights and non-quadratic potentials to capture higher-order residue dependencies.
Source: Definition · p. 4, Probability of a sequence · p. 5, Learning · p. 5Results
RBM features capture residue contacts, WW ligand-specific patterns, and Hsp70 subfamily differences; conditional sampling also yields diverse lattice sequences with high fitness.
Source: Inferred weights and interpretations · p. 6, Inferred weights and interpretation · p. 9, Hsp70 Protein · p. 13Takeaways & Limitations
Conditional sampling can generate diverse lattice sequences with fitness comparable to, or higher than, MSA sequences when using duplicated RBMs.
Source: Benchmarking on Lattice Proteins · p. 20The WW analysis does not clearly distinguish Type II from Type III domains, which often have high affinity for both ligand types.
Source: Inferred weights and interpretation · p. 9
Abstract
from arXiv · showhide
Statistical analysis of evolutionary-related protein sequences provides insights about their structure, function, and history. We show that Restricted Boltzmann Machines (RBM), designed to learn complex high-dimensional data and their statistical features, can efficiently model protein families from sequence information. We here apply RBM to twenty protein families, and present detailed results for two short protein domains, Kunitz and WW, one long chaperone protein, Hsp70, and synthetic lattice proteins for benchmarking. The features inferred by the RBM are biologically interpretable: they are related to structure (such as residue-residue tertiary contacts, extended secondary motifs ($\alpha$-helix and $\beta$-sheet) and intrinsically disordered regions), to function (such as activity and ligand specificity), or to phylogenetic identity. In addition, we use RBM to design new protein sequences with putative properties by composing and turning up or down the different modes at will. Our work therefore shows that RBM are a versatile and practical tool to unveil and exploit the genotype-phenotype relationship for protein families.
Results
The Results examine how RBMs represent protein-family sequences and learn interpretable structural, functional, and evolutionary patterns across Kunitz, WW, and Hsp70. They also assess contact prediction and sequence design, including how hidden-unit potentials and regularization shape model performance and representations.
Probability of a sequence
- Non-quadratic hidden-unit potentials let RBMs represent high-order residue dependencies beyond pairwise models.The paper presents RBMs as a framework for extracting sequence representations and modeling complex residue interactions.Source: Probability of a sequence · p. 5
Learning
- Sparse-weight penalties promote interpretable, compositional representations in which each sequence activates a subset of hidden units.The weights and local potentials are learned by maximizing average log-probability on the MSA, with model sampling used for gradient estimates.Source: Learning · p. 5
Description
- Kunitz hidden units identify residue contacts, including alternative electrostatic interactions at sites 45–49 and a 11–35 disulfide bridge or compensating charge interactions.A separate mode tracks Bikunin-related sequences, while other modes occur in evolutionarily distant sequences.Source: Inferred weights and interpretations · pp. 6–7, Figure 2 · p. 8
- Kunitz hidden unit 4 is associated with trypsin-like protease inhibition, while other modes may encode different specificities.Its site-12 basic residues and activity association agree with mutagenesis findings [1]; [2]; [3]; [4], while hydrophobic site-12 residues may indicate other serine-protease specificities Appel (1986).Source: Inferred weights and interpretations · pp. 6–7
Inferred weights and interpretation
- WW hidden units capture a contact between sites 4–22 and a beta-sheet-related feature across the beta1 and beta2 strands.The beta-sheet feature distinguishes mostly hydrophobic from hydrophilic residues and suggests solvent exposure in most, but not all, natural sequences.Source: Inferred weights and interpretation · p. 9, Figure 3 · p. 11
- WW hidden-unit activities correlate with ligand types: they distinguish Type I and IV from Type II/III, but do not clearly separate Types II and III.Specificity-related motifs agree with experimental findings on mutations and loop length, including Type IV preference for a longer, more flexible loop [1]; [2]; [3].Source: Inferred weights and interpretation · pp. 9–10, Figure 3 · p. 11
Hsp70 Protein
- Hsp70 hidden units distinguish families by NBD loop length, which is linked to nucleotide-exchange-factor interactions and functional diversification.Long, intermediate, and short loops are associated respectively with GrpE, Bag-1, and neither; mitochondrial and chloroplast Hsp70s can share the long-loop pattern with DnaK.Source: Hsp70 Protein · pp. 12–13
- Hsp70 modes also identify a candidate substrate-specificity feature in the SBD and capture coevolution between the SBD and LID.The residues supporting the interdomain mode are physically contiguous in the ADP conformation but not in the ATP conformation.Source: Hsp70 Protein · p. 13
- Hsp70 hidden units captured an inter-domain mode associated with allostery and a C-terminal-tail mode consistent with cochaperone interaction and possible partner recognition.Weight 4 inputs separate non-allosteric Hsp110 from other subfamilies; RBM also extracted multiple meaningful collective modes where sector analysis retrieved one sector.Source: Hsp70 Protein · pp. 13–14
Sequence Design
- RBM conditioning generated WW and Kunitz sequences combining features absent together in natural sequences, as well as sequences distant from natural ones with comparable probabilities.For WW, conditioning produced hybrid features; for Kunitz, it combined absence of the 11-35 disulfide bond with the Bikunin-AMBP feature. Duplicating hidden units can enable designed sequences with probabilities exceeding those in the MSA.Source: Figure 5 · p. 16, Sequence Design · p. 17
Contact Predictions
- RBM contact predictions were comparable to state-of-the-art direct-coupling methods on Kunitz and WW domains and across 17 families, though training cost was substantially higher.Performance increased with more hidden units and worsened under stronger weight-sparsifying regularization; the blind 17-family comparison used M = 0.3 R and λ2_1 = 0.1.Source: Contact Predictions · pp. 17–18, Figure 6 · p. 19
Benchmarking on Lattice Proteins
- In lattice proteins, RBM learned structural and negative-design features, and conditional sampling produced diverse sequences with fitness comparable to or higher than the training sequences.Fitness is the exactly computable probability of folding into the native structure; duplicated RBM could produce higher fitness than sequences in the MSA.Source: Benchmarking on Lattice Proteins · p. 20, Figure 7 · p. 21
Cross-validation of the model and interpretability of the representations
- The dReLU potential consistently outperformed quadratic and Bernoulli alternatives, while increasing hidden units improved performance until overfitting began.Mild regularization avoided overfitting and maximized test-set log-probability; stronger regularization made weights sparser with a small generalization loss, but excessive regularization reduced predictive power. Sparse weights promoted compositional representations in which units encode limited sequence portions and can be recombined for sequence design.Source: hidden-unit potentials · p. 22
hidden-unit potentials.
- Across protein families, RBM weights encoded contacts, structural residue groups, loop and gap patterns, binding-related motifs, and phylogenetic properties.Structural features included pairwise contacts, triplets in tertiary or secondary structure, and larger motifs such as those between β-strands; nonstructural features included sequence-length variation, loop length, binding specificity, and phylogenetic grouping.Source: hidden-unit potentials · pp. 22–23
C D E
The paper presents RBMs as a versatile framework for learning protein sequence features and generating sequences, while discussing their training, interpretation, and use across protein families. Its methods combine regularized learning and sampling-based procedures, with open questions about experimental validation and model scaling.
C D E
- Figures 8–10 show regularization effects and representative weights across protein families, including weights for families selected from Ekeberg et al. (2014).The Figure 8 caption contrasts no regularization and a test-optimal value of 0.03; the displayed Figure 3 weights use stronger regularization.Source: Figure 8 · p. 24, C D E · pp. 24–26, Figure 9 · p. 25
Discussion
- RBMs reveal structural, functional, and evolutionary features and can generate new sequences by tuning hidden units as representation-controlling knobs.The authors report studying 20 protein families, with detailed results for four families and 16 additional families suggesting motifs may be widespread.Source: Discussion · p. 27
- Sparsifying regularization improves weight interpretability and helps prevent overfitting, while dReLU potentials outperform other tested potentials for generative purposes.Increasing the number of hidden units improves likelihood up to a point, after which overfitting starts.Source: Discussion · p. 27
- RBM training requires intensive sampling, and the hidden-unit count needed to uncover all features of complex proteins such as Hsp70 remains unclear.The authors also say the biological significance of some inferred features and the functionality of designed sequences need experimental investigation.Source: Discussion · pp. 27–28
Data preprocessing; Objective function and gradients
- Kunitz and WW alignments are filtered to remove insertion columns and duplicate sequences, then reweighted using sequence identity; training maximizes average log-probability minus regularization costs.The custom L2/L1 weight penalty adaptively promotes homogeneity among hidden units and discourages units from becoming entirely disconnected.Source: Data preprocessing · p. 28, Objective function and gradients · p. 29
Moment evaluation; Stochastic Gradient Ascent
- Persistent Contrastive Divergence reuses prior Markov-chain samples with about 10 Gibbs updates per gradient evaluation, while stochastic gradient ascent uses data and model minibatches.The authors use Parallel Tempering to speed mixing for larger proteins.Source: Moment evaluation · p. 29, Stochastic Gradient Ascent · pp. 29–30
Dynamic reparametrization; Gauge choice
- Dynamic reparametrization adjusts hidden-unit slope and offset to keep activity near zero mean and unit variance; zero-sum gauges are enforced during training.Weight updates require reapplying the gauge transformation after each gradient update.Source: Dynamic reparametrization · p. 30, Gauge choice · p. 30
Evaluating the partition function; Sampling procedure; Computational complexity; Contact map estimation
- Annealed Importance Sampling estimates the partition function through ratios along interpolating distributions, and duplicated RBMs bias sampling toward high-probability sequences.The reported algorithm has complexity of order M × N × B and was tested for proteins up to about 700 sites; effective couplings for contact prediction are based on averaged double-mutation epistasis.Source: Evaluating the partition function · p. 30, Computational complexity · p. 31, Sampling procedure · p. 32
Supporting files
The supporting files provide weight logos and associated structures for selected protein-family models.
Supporting files
- Supporting files provide weight logos for Kunitz, WW, lattice-protein, and Hsp70 models, plus logos and structures for selected weights across 16 additional domains.They also include the 10 highest-norm sparse weights for each of those additional domains.Source: Supporting files · p. 33
Lattice-protein synthetic sequences
The lattice-protein benchmark uses synthetic sequences designed to fold into a specified structure, with correlations and higher-order interactions resembling features of real multiple-sequence alignments.
Lattice-protein synthetic sequences
- Lattice proteins are 27-residue chains whose possible folds are self-avoiding paths on a 3 × 3 × 3 lattice cube.The passage reports 103,406 possible folds.Source: Lattice-protein synthetic sequences · p. 38
- The benchmark contains 36,000 sequences generated to fold into structure S_A with probability P_nat(v; S_A) > 0.995.The sequences were generated by Monte Carlo simulations.Source: Lattice-protein synthetic sequences · p. 38
- The sequence data contain short- and long-range correlations as well as higher-order interactions arising from competition between folds.The contact map marks nearest-neighbor sites on the lattice, and pairwise amino-acid energy uses the Miyazawa–Jernigan potential Miyazawa and Jernigan (1996).Source: Lattice-protein synthetic sequences · p. 38
Model selection
Model selection balances generative performance with interpretable representations. The analyses favor dReLU potentials and sparsifying regularization, while showing that added hidden units improve fit only up to a point.
Number of hidden units
- Increasing hidden-unit count improves fit until overfitting or saturation; with regularization, likelihood saturates at about 100 hidden units on lattice-protein data.At larger counts, weights stabilize and new units can duplicate existing ones.Source: Model selection · pp. 38, 40
Sparse regularization
- Sparsity does not arise naturally in protein-sequence RBM training, and sparse weights improve interpretability while regularization can reduce generative performance.The selected regularization strength is presented as a compromise between these aims.Source: Model selection · p. 40
- L2^1 regularization yields sparse weights without disconnecting hidden units at high penalty, making it more robust and less dependent on fine-tuning than L1 regularization.Likelihood on the test set decreases mildly with increasing regularization strength.Source: Model selection · pp. 41–42
Hidden-unit potentials
- dReLU RBMs outperform Gaussian and Bernoulli RBMs in generative performance on lattice-protein and WW data at the same parameter count.The same pattern was reported for Kunitz; dReLU also models outlier sequences better than Gaussian potentials.Source: Model selection · pp. 42–43
- Bernoulli RBMs require more hidden units than other potential types to represent features, and encode more gap-stretch units than quadratic or dReLU RBMs.For 100-unit models, gap-stretch counts are 51, 23, and 15 on lattice proteins and 18, 15, and 9 on WW, respectively, for Bernoulli, quadratic, and dReLU potentials.Source: Model selection · p. 42
- dReLU potentials better model rare outlier sequences without altering the likelihood distribution for the bulk of sequences.The comparison used Kunitz-domain RBMs with 100 hidden units and the same stated regularization strength.Source: Model selection · pp. 42–43
- Contact features are comparatively robust to potential choice, while larger collective modes differ: Bernoulli representations are more redundant and Gaussian RBMs tend to miss outlier features.The passage attributes contact robustness to hidden units connected to only two sites producing pairwise effective interactions.Source: Model selection · pp. 42–43
Summary
- The study concludes that increasing hidden-unit count captures more features and improves generative performance until additional units duplicate existing ones without improving performance.The summary also reports that suitable sparsity penalties can retain both generative performance and interpretability.Source: Model selection · p. 43
Sequence generation
The paper evaluates RBM-generated sequences on synthetic lattice proteins and examines generated-sequence diversity for real protein families. On lattice proteins, inferred log probability predicts true fitness, while real-protein sequence quality cannot be assessed numerically because ground-truth fitness is unavailable.
Sequence generation
- RBMs with low likelihood generate low-quality lattice-protein sequences, while unregularized models can yield higher fitness but lower diversity.Conditional sampling also generates high-quality sequences when conditioned on unseen feature combinations.Source: Sequence generation · p. 44
- For lattice proteins, inferred log probability predicts the true fitness function, including on sequences not used for training.Generated-sequence quality and diversity vary with model capacity and regularization; regularized RBM samples better reproduce the data distribution than unregularized Boltzmann-machine samples.Source: Sequence generation · p. 44
- For real-protein RBMs, generated-sequence quality cannot be assessed numerically because ground-truth fitness is unavailable.Generated sequences, including those with recombined features absent in nature, are reported as consistent with a pairwise model trained on the same data.Source: Sequence generation · pp. 44–45
- Unregularized WW-domain models generate sequences closer to training data and have lower entropy than regularized models, with about 10^12 versus 10^18 distinct sequences, respectively.Sequence reweighting similarly increases entropy and makes generated sequences slightly farther from the training set.Source: Sequence generation · p. 45
Contact predictions
RBMs predict protein contacts through learned epistatic interactions, with performance depending on hidden-unit number, potential type, and weight sparsity.
Contact predictions
- Increasing hidden-unit number significantly improves contact prediction, and dReLU or quadratic hidden-unit potentials perform best.Source: Contact predictions · p. 46
- Contact performance strongly worsens as weight sparsity increases, while large sparse regularization and many hidden units reproduce pairwise-model performance on Kunitz and WW.Source: Contact predictions · pp. 47, 49
- Kunitz RBMs predict distant contacts well, though slightly worse than DCA, and their effective couplings agree with pairwise-model couplings.The contact-map comparison includes the top 551 RBM predictions and evaluates distant contacts with |i-j| > 4.Source: Contact predictions · pp. 46, 48
- On lattice proteins, quadratic and dReLU RBMs with many hidden units typically match pairwise models trained by Monte Carlo or pseudo-likelihood.Source: Contact predictions · p. 48
Feature robustness
WW features remain quite similar when training uses only one half of the sequence data.
Feature robustness
- Training on either half of the WW sequences yields closest features quite similar to the original features.Source: Feature robustness · p. 49
Comparison with the Hopfield-Potts model
The Hopfield-Potts model is a restricted RBM variant whose contact predictions and sequence generation are compared with RBM results.
Comparison with the Hopfield-Potts model
- The model lacks high-order interactions and a compositional regime, and it is not generative or able to reproduce data moments or generate diverse sequences.Source: Comparison with the Hopfield-Potts model · p. 51
- Hopfield-Potts contact predictions perform worse than RBM or plmDCA, while its high-eigenvalue collective-mode weights extend across most protein sites.Source: Comparison with the Hopfield-Potts model · p. 51
- With a small pseudo-count, Hopfield-Potts generated lattice sequences have very low folding probability; a larger pseudo-count gives reasonable but lower fitness than high-P(v) RBM sequences and low diversity.Source: Comparison with the Hopfield-Potts model · p. 51
- The lattice-protein benchmark is optimistic for Hopfield-Potts because the pseudo-count trick works less well for sequences with many conserved sites.Source: Comparison with the Hopfield-Potts model · p. 51
Additional figure: hidden-input distribution for Kunitz domain, separated by phylogenetic identity and genes
Kunitz hidden-unit features are examined against phylogenetic identity and gene-specific input distributions.
Additional figure: hidden-input distribution for Kunitz domain, separated by phylogenetic identity and genes
- Most Kunitz sequences lacking the disulfide bridge come from nematodes, and a sequence logo suggests an electrostatic triangle may replace the bridge.Source: Additional figure: hidden-input… · p. 56
- Hidden-input distributions are shown for Kunitz domains belonging to specific genes.Source: Additional figure: hidden-input… · p. 57
Additional figure: Weight logos, 3D visualizations, input distributions of 10 hidden units for Hsp70
Ten selected Hsp70 hidden units are labeled with features spanning loop variation, domain/function distinctions, subfamily identity, tail composition, and dimer contacts.
- Units 1, 6, and 7 label short-versus-long loop variation, including a long-loop variant.Source: Additional figure: Weight logos, 3D… · p. 57
- Units 2–4 and 9 label SBD function, interdomain features, and non-allosteric-specific patterns.Source: Additional figure: Weight logos, 3D… · p. 57
- Units 5, 8, and 10 label an unstructured tail, ER-specific identity, and dimer contacts.Source: Additional figure: Weight logos, 3D… · p. 57
Appendix 1 Figure 22. Truncated Weight logo of 10 selected HSP70 hidden units (2/2)
The Hsp70 hidden units distinguish subfamilies and tail-composition patterns, while corresponding structures show nucleotide-state conformations and dimer contacts.
Appendix 1 Figure 22. Truncated Weight logo of 10 selected HSP70 hidden units (2/2)
- The corresponding structures show ADP- and ATP-bound conformations, with dimer contacts highlighted for the last hidden unit.Source: Appendix 1 Figure 22. Truncated Weight… · pp. 60–62
- Hidden units 4 and 9 distinguish the non-allosteric subfamily, while unit 8 distinguishes ER-expressed eukaryotic Hsp70.Source: Appendix 1 Figure 22. Truncated Weight… · p. 63
Appendix 1 Figure 27. Some scatter plots of inputs for the 10 hidden units shown.
- A random-length null model with independently sampled tail amino acids differs from observed tail statistics, supporting a collective enrichment mode.Source: Appendix 1 Figure 27. Some scatter… · p. 65