ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing

Mingrui Li; Sixian Shen; Minzhang Li; Ruiyi Zhang; Kexin Zhang; Jiakai Zhang; Jingyi Yu

Source-linked AI summary

ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing

Mingrui Li, Sixian Shen, Minzhang Li, Ruiyi Zhang, Kexin Zhang, Jiakai Zhang, Jingyi Yu

arXiv:2609.04793v1Sep 4, 2026cs.AI

TL;DR

Dense PLM scaling raises computational costs without consistently improving mutation-sensitive prediction, while single-sequence models have limited explicit reuse of recurring local contexts. ProtLingo augments a pretrained backbone with centered latent N-gram memory and sparse expert routing. It achieves strong performance at the 150M scale across sequence-function and contact tasks while preserving structure-relevant representations.

Problem
Dense Transformer scaling increases computational cost without consistently improving downstream performance, while single-sequence PLMs have limited explicit mechanisms for reusing recurring local contexts.
Method
ProtLingo combines centered latent N-gram memory, which retrieves local-context residuals, with sparse expert routing in a pretrained ESM2 backbone.
Results
ProtLingo achieves strong performance at the 150M scale across sequence-function prediction, contact prediction, and language modeling while preserving structure-relevant representations.
Takeaways & Limitations
Conditional memory and expert routing provide effective alternatives to substantially enlarging dense protein language models.
Takeaways & Limitations
Gains over ESM2-150M are bounded by the lightweight current modules, suggesting richer routing designs may unlock further conditional specialization.

Abstract

from arXiv · show

Proteins perform diverse cellular functions, and even single amino-acid substitutions can alter stability, activity, or molecular interactions. Protein language models (PLMs) provide a scalable approach for modeling such sequence--function relationships from unlabeled sequences, but increasing the size of dense Transformer backbones often brings substantial computational cost without consistently improving mutation-sensitive prediction. We introduce ProtLingo, an efficient PLM framework that augments a pretrained single-sequence backbone with conditional local memory and sparse expert routing. ProtLingo maps contextual residue representations into route-specific discrete codes, composes centered local windows into latent $N$-gram addresses, and retrieves reusable residual signals associated with recurring local sequence contexts. In parallel, selected feed-forward blocks are upcycled into sparse Mixture-of-Experts layers with shared and routed experts, enabling residue-dependent computation while activating only a subset of parameters. Experiments on protein fitness prediction, FLIP benchmarks, and supervised contact prediction show that ProtLingo achieves competitive performance with a 150M-scale backbone, including strong parameter efficiency on mutation-effect prediction and preserved long-range structural representations.

1 Introduction

ProtLingo addresses the cost and limited mutation sensitivity of dense PLMs by adding conditional local memory and sparse expert routing to a pretrained backbone. Across fitness, clinical variant, FLIP, and contact prediction, it achieves competitive performance at the 150M scale while retaining structure-relevant representations.

Dense Transformer scaling increases computational cost without consistently improving downstream performance, motivating more efficient protein language models.
153M active parameters support competitive DMS prediction, matching or exceeding several larger dense baselines while using fewer active parameters than ESM2-650M and ESM-1b.
Single-sequence PLMs need better ways to exploit recurring local contexts because mutation effects often depend on nearby conserved residues.
ProtLingo maps contextual residues to discrete codes, composes centered local windows into latent N-gram addresses, and retrieves reusable residual signals.
ProtLingo combines conditional memory with sparse expert routing to reuse local contexts and allocate computation adaptively across residue environments.
Across protein fitness, clinical variant, FLIP, and supervised contact prediction, ProtLingo remains competitive at the 150M scale and preserves comparable contact precision to ESM2-150M.

2 Related Work

Protein language models learn transferable biological representations from unlabeled sequences, but their distributed backbones provide limited explicit reuse of recurring local patterns. Conditional computation offers a complementary route to greater capacity without activating the full model for every residue.

Transformer-based PLMs such as ProtTrans, ESM-1b, ESM2, and XTrimoPGLM learn contextual residue representations through large-scale self-supervised pretraining.
Downstream gains are not monotone in model size, with variant-effect performance saturating beyond mid-scale regimes.
Distributed PLM representations offer limited explicit mechanisms for reusing recurring local motifs, conserved contexts, and short functional segments.
Sparse Mixture-of-Experts routing decouples total model capacity from per-token active computation by assigning each token to a small subset of experts.

3 Method

ProtLingo adapts a pretrained protein language model with centered latent N-gram memory and sparse expert routing. These modules add motif-conditioned residual retrieval and residue-dependent computation while retaining the pretrained backbone and enabling downstream structural and sequence-function applications.

Architecture: ProtLingo combines centered latent N-gram memory with sparse expert routing to adapt a pretrained protein language model.The architecture inserts memory before selected Transformer blocks and replaces selected feed-forward blocks with sparse experts.
Backbone and layer-wise adaptation: The backbone produces hidden states H(0) through H(B), while conditional memory and MoE modules are inserted at selected layers.H(0) is the embedding output, H(ℓ) is the representation after ℓ Transformer blocks, and B is the number of blocks.
Centered latent N-gram memory: Centered windows Wi = (i − s, ..., i, ..., i + s) capture local context for motif- and mutation-sensitive modeling.The window size is N = 2s + 1.
Centered latent N-gram memory: ProtLingo projects residue representations into route-specific binary codes, combines windowed symbols into centered latent N-gram addresses, and retrieves route-specific memory vectors.Each residue produces R route-specific symbols, and each address indexes a corresponding memory table.
Memory retrieval and injection: Retrieved memory is compressed, projected into keys and values, gated by representation-key similarity, locally refined, and injected as a residual before selected Transformer blocks.Zero-initialized memory and convolution weights make the initial memory residual zero, preserving the pretrained function while residual corrections are learned.
Sparse expert adaptation: MoE upcycling replaces selected dense feed-forward blocks with one shared expert and multiple routed experts, using top-1 routing and auxiliary losses for expert usage and router-logit regularization.The masked language modeling objective jointly updates the backbone and memory parameters during continuation pretraining.

4 Experiments

ProtLingo is evaluated across mutation-effect, fitness, language-modeling, and long-range contact tasks, emphasizing parameter efficiency and representation preservation. The experiments compare its compact active-parameter budget with dense baselines and assess contributions from conditional modules.

Evaluation scope: Evaluation covers protein fitness, clinical variant classification, FLIP benchmarks, supervised contact prediction, and intrinsic language modeling.Metrics include ProteinGym DMS Spearman correlation and AUC, FLIP task scores, long-range contact precision, MLM loss, and perplexity.
Supervised contact prediction: The contact protocol freezes each encoder and trains the same lightweight head on pairwise residue features for long-range residue pairs.Contacts use a Cβ–Cβ distance threshold below 8 Å, with Cα substituted for glycine, and require |i−j| ≥24.
Supervised contact prediction: 0.485, 0.657, and 0.816 are ProtLingo’s P@L, P@L/2, and P@L/5 contact scores with 153M active parameters.It slightly improves over ESM2-150M at P@L and matches it at P@L/2; the experiment is a representation-preservation diagnostic.
Fitness and mutation-effect prediction: ProtLingo improves over ESM2-150M on ProteinGym Spearman and AUC while remaining close to the 150M-scale baseline on FLIP tasks.ProtLingo attains the highest parameter-efficiency score, defined as ProteinGym DMS Spearman divided by active parameters in billions.
Ablation study: Ablations remove sparse MoE, motif-aware Lngram memory, or both, comparing changes in ProteinGym DMS Spearman correlation and language-model perplexity against full ProtLingo.Higher ∆ρ and lower ∆PPL indicate better performance under the reported convention.

5 Model Analysis

Model analysis finds that latent memory captures recurring motif structure and that sparse experts specialize across biological annotations. These analyses connect motif-sensitive residual changes with differentiated routing rather than globally interchangeable experts.

Motif-aware representations: 54.2% of motif windows shared the same layer-21 address versus 23.4% of matched non-motif controls.Across 10,447 motifs, address concentration was compared with amino-acid-composition-matched windows from the same proteins.
Motif-aware representations: 0.968 within-family residual cosine similarity exceeded 0.756 between-family similarity across 10,573 occurrences from 10 PROSITE families.The analysis also found similar route-local address combinations within motif families while retaining protein-context variation.
Sparse expert specialization: Rank-1 expert selection outperformed lower-ranked alternatives, whose use reduced ProteinGym and FLIP correlations and increased MLM loss.Figure 4 compares performance under Rank-1 through Rank-4 expert selection.
Sparse expert specialization: Mean normalized routing entropy was 0.971, corresponding to 3.85 effective experts and indicating no global expert collapse across 27 MoE layers.The analysis covered 4.23 billion Top-1 decisions across 461,859 deduplicated proteins.
Sparse expert specialization: 80–99% of residues in specific biological categories were routed to one expert at particular layers, revealing multiscale specialization across structures, functional residues, and conserved domains.The routing analysis controlled for amino-acid and positional biases using matched residues from the same proteins.

6 Discussion

ProtLingo combines centered latent N-gram memory with sparse expert routing to improve mutation-sensitive modeling without substantially larger dense models. At the 150M scale, it delivers strong performance while preserving structure-relevant representations, though gains over ESM2-150M remain bounded by the lightweight modules.

Gains over ESM2-150M are bounded by the lightweight nature of the current modules.The authors suggest richer routing designs as a direction for further specialization.
ProtLingo combines centered latent N-gram memory with sparse expert routing in a pretrained ESM2 backbone.The framework introduces reusable local-context memory and residue-dependent computation.
ProtLingo improves mutation-sensitive modeling without relying on substantially larger dense models.This conclusion positions conditional memory and expert routing as alternatives to continued dense scaling.
Strong performance at the 150M scale is achieved while preserving structure-relevant representations.The reported experiments span sequence-function prediction, contact prediction, and language modeling.

7 Supplementary Details

The supplementary details specify ProtLingo’s architecture, training and evaluation procedures, while ablations and routing analyses characterize its memory contribution, sampling limitation, and layer-dependent expert specialization.

Model Architecture: ProtLingo continues pretraining from ESM-2 150M, retaining dense blocks 0–2 and converting blocks 3–29 into MoE layers.The backbone uses 30 Transformer blocks, hidden size 640, 20 attention heads, and feed-forward width 2,560.
MoE Architecture: Each token activates one shared expert and one top-1 routed expert, preserving the original active feed-forward width of 2,560.Four routed experts are initialized from one partition of the pretrained feed-forward channels, while the other partition initializes the shared expert.
LNgram Architecture: LNgram modules use centered three-token windows and route-specific codes to form 125 latent memory addresses.Modules are inserted before blocks 1, 11, and 21, with 320 routes, memory dimension 8, and latent dimension 256.
Training Configuration: Training uses 40,000 optimizer steps on NVIDIA ESM-2 UniRef data, with 786,432 tokens per optimizer step and 15% MLM corruption.The step-40k checkpoint is used for downstream evaluations, with BF16 computation and an FP32 exponential moving average.
Benchmarking and Evaluation Metrics: Evaluation covers ProteinGym, FLIP, and CASP15 using Spearman correlation, AUC, and long-range contact precision metrics.ProteinGym uses masked-marginal mutation scores and nested aggregation across five functional categories.
Ablation Studies: Removing LNgram degrades MLM perplexity and downstream mutation-effect prediction, supporting complementary information from motif memory.The authors also report that the 80/20 mixed sampling strategy remains a practical working hypothesis without a fully controlled ablation.