Source-linked AI summary
EvoFlows: Evolutionary Edit-Based Flow-Matching for Protein Engineering
Nicolas Deutschmann, Constance Ferragu, Jonathan D. Ziegler, Shayan Aziznejad, Eli Bixby
TL;DR
Protein optimization needs models that can make a controlled number of local changes, including insertions and deletions, without specifying mutation sites. EvoFlows learns edit-based transitions between related sequences, and evaluations report competitive family-level modeling with more diverse natural-like variants. The paper’s evaluation is in silico, so functional fitness still requires wet-lab validation.
Problem
Existing protein language models are poorly aligned with local optimization: autoregressive generation regenerates full sequences, while masked and diffusion approaches require specified mutation sites and do not naturally handle indels.
Source: §1 · pp. 1–2Method
Results
EvoFlows captures family-level sequence distributions competitively with state-of-the-art baselines while generating more diverse variants without externally specified mutation sites.
Source: §1 · p. 2Takeaways & Limitations
Across six diverse protein families, EvoFlows generates non-trivial, natural-like variants that explore larger mutational neighborhoods.
Source: §5 · p. 10Evaluation is limited to in silico analysis, and wet-lab validation remains essential to confirm functional fitness.
Source: §5 · p. 10
Abstract
from arXiv · showhide
We introduce EvoFlows, a variable-length protein sequence-to-sequence modeling approach designed for protein engineering. Existing protein language models are poorly suited for optimization tasks: autoregressive models require full sequence generation, masked language and discrete diffusion models rely on pre-specified mutation locations, and no existing methods naturally support insertions and deletions relative to a template sequence. EvoFlows learns mutational trajectories between evolutionarily related protein sequences via edit flows, allowing it to perform a controllable number of mutations (insertions, deletions, and substitutions) on a template sequence, predicting not only _which_ mutation to perform, but also _where_ it should occur. Through extensive _in silico_ evaluation on diverse protein families from UniRef and OAS, we show that EvoFlows generates variants that remain consistent with natural protein families while exploring farther from template sequences than leading baselines.
1 Introduction
Protein optimization seeks targeted mutations that improve an existing protein while preserving its structure and function. EvoFlows models local sequence edits, including insertions and deletions, and the paper reports competitive family-level modeling with more diverse variants and no externally specified mutation sites.
1 Introduction
- Protein optimization seeks targeted substitutions, insertions, and deletions that improve an existing protein while preserving overall structure and function.Unlike de-novo design, optimization starts from a known functional protein and explores its local neighborhood.Source: §1 · p. 1
- Autoregressive models regenerate entire proteins, while masked language models require specified substitution sites and do not naturally support insertions, deletions, or variable lengths.These constraints make existing PLMs poorly aligned with local-mutation optimization.Source: §1 · p. 1
- Fit-for-purpose optimization models should support local edits without externally specified mutation locations and handle insertions and deletions, which are common in antibody families.Such models should generate function-preserving variants for unguided exploration before task-specific conditioning.Source: §1 · p. 2
- EvoFlows captures family-level sequence distributions competitively with state-of-the-art baselines while generating more diverse variants without externally specified mutation sites.The approach learns mutation-based sequence-to-sequence transition rules and supports variable-length sequences.Source: §1 · p. 2
- The paper also proposes a grid-free inference procedure intended to better suit vectorized operations.Source: §1 · p. 2
2 Related work
Related work describes how protein language models generate sequences and why their generation paths constrain local optimization. It also introduces evotuning and discrete flow matching, including edit flows that support elementary edits and variable-length sequences.
2.1 Protein Language Models (PLMs)
- Autoregressive PLMs generate entire sequences, making mutation count and location difficult to control for tasks requiring only a few targeted changes.Even approaches conditioned on context sequences still generate the full sequence to obtain a few mutations.Source: §2.1 · p. 3
- Masked PLMs produce residue-level conditional distributions, so generation requires choosing mask locations and assumes a fixed sequence length.Source: §2.1 · p. 3
- Discrete diffusion models learn to reverse stochastic corruption, but optimization from a starting sequence still requires selecting corruption locations, while fully corrupted starts entail unnecessary computation and conditioning.The diffusion framework generalizes masked and autoregressive generation paths.Source: §2.1 · pp. 3–4
2.2 Evotuning
- Evotuning adapts a pretrained PLM to a protein family using the same self-supervised objective on homologous sequences, and can improve family-specific constraints and downstream prediction or design.The passage argues that foundational PLMs should be evotuned when compared with family-specific models.Source: §2.2 · p. 4
2.3 Discrete Flow Matching (DFM) and Edit Flows
- Discrete flow matching transports source sequences toward target sequences by learning local flows over sequence pairs.Source: §2.3 · p. 4
- Edit flows extend transitions to substitutions, insertions, and deletions, using aligned sequence pairs with an empty-character token to represent indels.The model is trained over time to predict elementary edits, with a tractable sum because flow is nonzero only for sequence pairs differing by one edit.Source: §2.3 · p. 4
- Edit flows combine local sequence modification with variable-length generation, avoiding ad hoc mask insertion and reflecting how single-amino-acid mutations connect sequences.Source: §2.3 · p. 5
3 EvoFlows
EvoFlows models transitions between homologous protein sequences through substitutions, insertions, and deletions. It uses aligned sequence pairs and continuous-time sampling to generate variable-length variants, with rate scaling for control over mutation count.
3 EvoFlows
- EvoFlows connects evolutionarily related protein sequences through substitution, deletion, and insertion operations.The model is trained over a distribution of aligned homolog pairs.Source: §3 · p. 5
3.1 Homologous protein pair distributions
- Homologous sequences approximate functional similarity, and database searches provide related natural sequences for the pairwise training distribution.The procedure searches databases such as UniRef30 and OAS for homologs of a starting protein.Source: §3.1 · p. 5
3.2 Pairwise protein alignment
- Needleman–Wunsch alignments augment protein sequences with ε tokens so paired sequences have equal length for edit-flow modeling.The resulting insertions and deletions correspond to sequence edits.Source: §3.2 · pp. 5–6
3.3 Inference sampling
- At inference, EvoFlows samples a continuous-time Markov chain to produce a variant, drawing the next event time from the model’s total rate before selecting a mutation.The sampling procedure replaces repeated Bernoulli sampling with a single uniform draw and avoids adding potentially large correction terms to prevent pathological distributions.Source: §3.3 · p. 6
- Length-normalized clock scaling controls the number of mutations independently of sequence length.This is introduced as a hyperparameter for inference.Source: §3.3 · p. 6
3.4 Architecture 𝑡(𝑥′
- The architecture uses a pretrained ESM-2 encoder, FiLM time conditioning, and shallow MLP heads to predict mutation rates.Applying FiLM late makes updating time for a fixed input negligible in cost, even when numerical integration requires many forward passes.Source: §3.4 · pp. 6–7
4 Results
EvoFlows is evaluated first on a deterministic mutation task and then on protein homolog datasets, with baseline comparisons and inference-runtime analysis. The reported comparisons show variants that stay close to the holdout distribution while exploring a larger local neighborhood than the baselines.
4 Results
- The study validates edit flows on a synthetic task with known expected mappings before evaluating EvoFlows on six protein homolog datasets.The homolog datasets are derived from seed proteins and compared with a holdout split.Source: §4 · p. 7
4.1 Deterministic setup
- On the deterministic task, position-dependent rules produce a unique target sequence, enabling precise assessment of mutation type, position, and amino-acid identity.Insertions are applied first in increasing index order, followed by deletions and then substitutions.Source: §4.1 · p. 7
4.2 EvoFlows
- The evaluation compares variants from six seed proteins with a holdout split, matching expected mutations per sequence across methods except random homolog pairing.Random inference homolog pairing provides a reference for similarity between natural homologs, while random mutations are a lower bound that does not preserve family-level structure.Source: §4.2 · pp. 7–8
- Figure 2 shows that clock normalization shifts precision-recall trade-offs for insertions, substitutions, and deletions, while no-op predictions remain stable.The benchmark uses known deterministic mutation patterns to assess model behavior.Source: §4.1 · p. 7, Figure 2 · p. 8
EvoFlows inference
- Training pairs are formed from sequence pairs globally aligned with Needleman-Wunsch after homolog collection and filtering.The homolog search uses UniRef30 profile search and an additional search of ColabFold’s environmental database.Source: EvoFlows inference · p. 8
4.3 Baselines
- EvoFlows stays close to the holdout distribution while exploring a larger local neighborhood; EvoDiff-MSA explores less and produces less diverse variants, and random mutations perform worst across metrics.Evotuning without forced substitutions can introduce as few as 1.5 mutations per sequence on average; forcing substitutions increases diversity but substantially degrades distributional quality.Source: Figure 3 · p. 9
- The MLM baselines support substitutions only, so comparisons involving them are limited to substitutions.The paper notes that no widely used method exists for generating indels with MLMs.Source: §4.3 · pp. 9–10
4.4 Inference runtime
- EvoFlows requires forward passes proportional to the expected number of edits, while autoregressive models require passes proportional to sequence length.EvoDiff-MSA repeatedly masks and regenerates candidate edit positions, including gap runs for insertions, with weaker control over final edit count.Source: §4.4 · p. 10
5 Conclusion
EvoFlows models protein transitions through substitutions, insertions, and deletions, and its evaluation spans six diverse protein families. Its variants are natural-like, explore larger mutational neighborhoods than leading baselines, and remain close to the holdout distribution; wet-lab validation is still needed to confirm functional fitness.
5 Conclusion
- EvoFlows learns protein transition rules through substitutions, insertions, and deletions, explicitly modeling these as edit operations in template-based optimization.The authors state that, to their knowledge, it is the first method to directly model insertions and deletions as explicit edit operations in this setting.Source: §5 · p. 10
- Across six diverse protein families, EvoFlows generates non-trivial, natural-like variants that explore significantly larger mutational neighborhoods than state-of-the-art baselines while remaining close to the holdout distribution.The families span enzymes, growth factors, and antibody fragments.Source: §5 · p. 10
- Evaluation is limited to in silico analysis, so wet-lab validation remains essential to confirm functional fitness.Source: §5 · p. 10
- EvoFlows learns a prior over plausible local sequence changes rather than directly optimizing particular engineering objectives.The authors identify train-time and inference-time conditioning as promising directions, including property guidance, sequence constraints, and metadata conditioning.Source: §5 · p. 10
A Architecture Details
EvoFlows embeds each input sequence with a pretrained ESM-2 model whose encoder is fine-tuned jointly with the model. Time conditioning modulates token representations, while edit-transition rates and token distributions are parameterized by shallow MLPs and ESM-2 prediction heads.
A Architecture Details
- Each sequence is embedded with pretrained ESM-2, including explicit start and end tokens, and the encoder is fine-tuned jointly with EvoFlows.Source: A Architecture Details · p. 13
- Time is embedded with a sinusoidal embedding and MLP, then used by FiLM to produce feature-wise scale and shift vectors for token representations.The time-conditioning design allows time to change efficiently for fixed inputs, useful when training with multiple time samples per sequence in a batch.Source: A Architecture Details · pp. 13–14
- The transition parameterization defines substitutions, insertions, and deletions as sequence edits at specified positions.The passage defines substitution and insertion by token and position, and deletion by position.Source: A Architecture Details · p. 14
- Shallow MLPs parameterize positional rates and token distributions, with rates normalized for positivity and numerical stability.Token distributions use ESM-2 language prediction heads.Source: A Architecture Details · p. 14
B Evaluation Metrics
The evaluation metrics section motivates metrics and distances suited to protein sequence generation, noting that generative-model evaluation is difficult. It frames dataset-level metrics and distances as ways to compare generated sequences with held-out natural sequences.
B Evaluation Metrics
- Generative-model evaluation is difficult, motivating metrics and distances selected for protein sequence generation.Source: B Evaluation Metrics · p. 14
- The section distinguishes metrics that characterize individual datasets from distances that compare synthetic sequences with held-out natural sequences from the training MSA.Source: B Evaluation Metrics · p. 14
- The Kullback-Leibler divergence is introduced as an information-theoretic measure of how one probability distribution differs from another.Source: B Evaluation Metrics · p. 14
B.1 Dataset Heuristics
The evaluation appendix describes sequence-level distribution heuristics, model-based pseudo log likelihood, and measures of coevolutionary covariance and mutual information. These metrics characterize generated datasets and compare them with natural sequence distributions.
B.1 Dataset Heuristics
- A well-performing unconditioned generator is hypothesized to match a subsampled hold-out MSA on sequence-length, amino-acid, and aligned per-position amino-acid and gap distributions.Source: §B.1 · p. 14
B.2 Model-Based Pseudo Log Likelihood
- For pseudo log likelihood, the cumulative likelihood of a generative path is computed; masked language models use one random order of individually masked positions.Autoregressive models have a single well-defined path, unlike masked language models.Source: §B.2 · pp. 14–15
B.3 Covariance and Mutual Information
- Covariance between aligned positions can indicate coevolution, including compensatory changes associated with maintaining structural contacts.The model should reproduce natural-family covariance patterns if it captures these evolutionary constraints.Source: §B.3 · p. 15
- The appendix defines four-dimensional covariance from positional and joint amino-acid frequencies, then contracts it into positional interaction strength using a Frobenius norm.Source: §B.3 · p. 15
- Mutual information with average product correction provides an interpretable two-dimensional representation of coevolution.Source: §B.3 · pp. 15–16
B.4 BLOSUM-Corrected KL Divergence
This section uses KL divergence to compare amino-acid frequency distributions, then applies BLOSUM62-informed smoothing to keep comparisons finite when observed frequencies are zero.
B.4 BLOSUM-Corrected KL Divergence
- KL divergence measures differences between amino-acid frequency distributions, but is asymmetric and can become infinite when q(a)=0 while p(a)>0.It is non-negative and equals zero only for identical distributions.Source: §B.4 · p. 16
- BLOSUM62 background frequencies serve as biologically informed pseudo-count priors for smoothing observed amino-acid frequencies.The prior frequencies reflect amino-acid abundance in protein databases and are used instead of uniform pseudo-counts.Source: §B.4 · p. 16
- The smoothed probability combines observed counts with a weighted prior over 21 amino-acid and gap types.The gap prior frequency was calibrated empirically to 0.7%.Source: §B.4 · p. 16
1. Ensures biologically plausible probability estimates even with limited data
The section describes biologically informed smoothing for limited data and spectrum-kernel MMD for comparing generated and reference sequence distributions.
1. Ensures biologically plausible probability estimates even with limited data
- BLOSUM-informed pseudo-counts weight amino acids by natural abundance, provide a Dirichlet-prior interpretation, and reduce variance while converging to maximum-likelihood estimates as data grow.The smoothed distances remain finite and meaningful across datasets with different sizes or sampling depths.Source: §1 · pp. 16–17
B.5 Spectrum Kernel Maximum Mean Discrepancy
- Spectrum-kernel MMD compares generated and reference distributions through k-mer features without requiring sequence alignment.The spectrum kernel represents sequences as k-mer count vectors and computes similarity through their inner product.Source: §B.5 · pp. 17–18
- MMD approaches zero for indistinguishable distributions, and a characteristic kernel makes zero MMD equivalent to identical distributions.The paper uses the biased estimator, which guarantees non-negative MMD^2 and has lower variance than the unbiased alternative.Source: §B.5 · p. 17
- The spectrum kernel is sparse-computation friendly, makes no assumptions about the data distribution, captures local composition, and is described as robust for artificial sequences.Its sparse representation avoids explicitly enumerating all |A|^k k-mers.Source: §B.5 · p. 18
C Protein Types and Datasets
The paper evaluates homolog datasets from six protein families and uses supplementary figures to compare mutation prediction, baseline behavior, and sequence-frequency profiles.
C Protein Types and Datasets
- The homolog datasets were constructed by iterative profile search using Anti-SARS-CoV-2 VHH (Ty1), serine-pyruvate aminotransferase, DhaA, anti-HER2 scFv, chicken FGF2, and Anti-EphA2 VHH as seed proteins.Source: C Protein Types and Datasets · pp. 18–19
- Table 2 reports homolog dataset statistics, with query and target coverage defined as the fractions of sequences aligned in each homolog pair.The supplied caption does not provide the table's numerical values.Source: Table 2 · p. 19, C Protein Types and Datasets · p. 19
C.1 Supplementary Figures
- Figure 4 shows mutation-type F1 across sequence lengths and a predicted-versus-ground-truth confusion matrix; accuracy remains stable, with residual errors mainly confused with no-op.The matrix categories are no-op, insertion, substitution, and deletion.Source: Figure 4 · p. 20
- Figure 5 compares datasets across metrics: EvoFlows variants remain close to holdout distributions, while forced-mutation evotuning degrades across metrics.The random baseline performs worst, and unforced evotuning introduces almost no mutations.Source: Figure 5 · p. 21
- Figure 7 places EvoFlows-generated and holdout per-position amino-acid frequency heatmaps side by side; their similar profiles indicate preservation of family-specific positional distributions.Rows are datasets and color intensity represents aligned per-position amino-acid frequency.Source: Figure 7
- Figure 6 plots per-dataset metric values for Random pairing, EvoFlows, Evotuning, forced-mutation Evotuning, and the random baseline.Each subplot represents an evaluation metric, with lines connecting values across datasets.Source: Figure 6 · p. 22