Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders

1. Introduction

• Sequence design is a costly black-box problem over discrete, high-dimensional inputs, with multiple objectives and scarce experimental labels.The paper frames BayesOpt as a way to account for uncertainty and the explore-exploit tradeoff when labels are scarce.Source: §1 · pp. 1–2
• The paper introduces a large-molecule in silico task that emulates protein design more closely than common open-source benchmarks.The task optimizes fluorescent-protein stability and SASA.Source: §1 · p. 2
• The new Pareto frontier for RFP stability and SASA is denser with sequences that improve on their ancestors.Figure 1 depicts ancestors as circles and optimized offspring as crosses; higher stability and SASA are preferred.Source: Figure 1 · p. 1, §1 · p. 2
• LaMBO combines a generative denoising autoencoder with a discriminative deep-kernel-learning Gaussian process in a jointly trained architecture.The approach also includes a decision-optimization routine.Source: §1 · p. 2
• LaMBO improves sample efficiency and solution quality against genetic and latent-space BBO baselines across two small-molecule tasks and a new large-molecule task.The evaluation also includes in vitro fluorescent-protein experiments.Source: §1 · p. 2
• In vitro, LaMBO discovers red fluorescent proteins that are brighter and more thermostable.The paper presents this as a wet-lab result.Source: §1 · p. 2

2.1. Discrete Multi-Objective Sequence Design

• Because the sequence space grows exponentially with length, optimization starts from a pool and chooses a base sequence, positions to change, and replacement tokens.The passage describes these as three nested decisions.Source: §2.1 · p. 3
• Multi-objective sequence design seeks non-dominated solutions rather than one universal optimum, and uses hypervolume to find candidates near the unknown Pareto frontier.The hypervolume is bounded by the candidate set and a reference solution.Source: §2.1 · p. 3

2.2. BayesOpt

• BayesOpt repeatedly fits a probabilistic surrogate, selects queries by optimizing an acquisition function, observes their outcomes, and retrains.GPs are often preferred in data-scarce regimes, while standard kernels can struggle with high-dimensional inputs.Source: §2.2 · p. 3

3. Related Work

• Genetic algorithms such as NSGA-II evolve solutions through random mutation, while other discrete approaches screen or propose mutations using learned models.The passage characterizes the discussed approaches as greedy when they select queries from point estimates of objective values.Source: §3 · p. 3
• Some discrete BayesOpt methods optimize directly in sequence space with substring kernels and genetic algorithms, often on small tasks.The passage also notes antibody design that uses task-specific knowledge of complementarity-determining regions.Source: §3 · p. 4
• Latent-space BayesOpt shares sequence embeddings between a generative decoder and a discriminative surrogate, while LaMBO adds joint training, avoids autoencoder pretraining, and supports multi-objective tasks and acquisitions.The cited LSBO approach uses a pretrained VAE and separately trains its VAE weights and GP head.Source: §3 · p. 4
• Multi-objective BayesOpt has established acquisitions such as NEHVI and multi-task GP surrogates, which LaMBO adopts for its sequence-design setting.The passage also identifies an efficient multi-task GP posterior-sampling approach used by the paper.Source: §3 · p. 4

4.1. Architecture Overview

• A non-autoregressive denoising autoencoder and multi-task GP jointly support latent sequence optimization, with NEHVI used for objectives that can be noisy.The GP operates on pooled sequence embeddings and supplies a posterior predictive distribution to the acquisition function; the approach does not require a pretrained autoencoder.Source: §4.1 · p. 4

4.2. Initializing the Latent Embeddings

• Corrupted, encoded variants of current Pareto solutions initialize latent optimization, reducing the chance that gradient search starts in a flat region unlikely to improve the solutions.Corruption selects positions and applies substitutions, insertions, or deletions; the operation depends on whether the decoder is an MLM or LANMT head.Source: §4.2 · p. 5
• Base sequences combine current Pareto solutions, earlier Pareto sequences, and history samples, with higher-scoring sequences more likely to be optimized.Weights use each sequence’s least favorable ranking across objectives; the base-set size matches the query batch size.Source: §4.2 · p. 5

4.3. Sequence Optimization

• The inner loop treats the decoder as a proposal distribution, iteratively refining latent embeddings along a regularized acquisition gradient and scoring sampled sequence batches.LaMBO retains the best-acquisition batch found during optimization rather than simply decoding once from the final latent embeddings.Source: §4.3 · p. 6
• An entropy penalty keeps decoder proposals non-uniform; unregularized acquisition gradients quickly raised entropy and made proposals resemble random search.The penalty also smooths the query-loss surface and can reduce dependence on a good initialization; choosing a step size alone was difficult.Source: §4.3 · p. 6

5.2. Comparing to Multi-Objective Genetic Optimizers

• LaMBO outperformed strong genetic baselines across the four tasks, particularly on natural-sequence tasks, while the high-entropy Bigrams start distribution slowed learning.The baselines include NSGA-2 and a model-based genetic optimizer using the same surrogate architecture and acquisition function.Source: §5 · p. 6, §5.2 · p. 7

5.1. Evaluation procedure

• Evaluation spans Bigrams, logP + QED, DRD3 docking + SA, and RFP Stability + SASA; tasks generally start with 512 examples and receive 1,024 online queries in batches of 16, without pretraining data.Methods are evaluated by relative hypervolume improvement over the starting Pareto frontier.Source: §5.1 · pp. 6–7

5.3. Analyzing LaMBO

• DAE-generated proposals improve performance across tasks, while proposal optimization helps most on Bigrams and the entropy penalty is detrimental there but helpful on biological sequences.Bigrams’ non-dominated solutions lie far outside its starting sequence distribution, so the entropy penalty can keep queries too near previously seen sequences.Source: Figure 4 · p. 7, §5.3 · p. 7

5.4. Analyzing Sequence Designs

• Across tasks, every point on the old Pareto frontier is strictly dominated by at least one point on the new frontier, whose solutions are evenly spread.The figure also reports wet-lab brightness and thermostability measurements for RFPs optimized for Stability + SASA.Source: Figure 5 · p. 8, §5.4 · p. 8

5.3. Analyzing LaMBO

• On RFPs, NEHVI produces slightly higher hypervolume than averaging normalized objectives: 1.57 versus 1.51, with more exploration of frontier extremes and diverse tradeoffs.Score averaging quickly pushes out the frontier interior but explores less.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4. Analyzing Sequence Designs

• Wet-lab testing found non-dominated variants for three of five RFP ancestors, including strictly dominant mScarlet and DsRed.M1 variants; results for two ancestors were inconclusive.The authors identify Stability + SASA as useful in-silico proxy objectives for this protein-design task.Source: §5.4 · p. 8, Figure 5 · p. 8
• Small-molecule sequences changed significantly along the Pareto frontiers despite no explicit sequence-diversity objective, offering interpretable differences among non-dominated designs.The paper connects such comparisons to insights about objective tradeoffs in drug design.Source: §5.4 · pp. 8–9
• Some proposed molecules contain large macrocycles or three-carbon rings likely to be synthetically inaccessible; logP + QED does not explicitly incentivize accessibility.The authors note that comparing frontier molecules can reveal such structural concerns even with little chemistry knowledge.Source: §5.4 · pp. 8–9

6. Discussion

• LaMBO achieved good sample efficiency and solution quality across design tasks, optimized red fluorescent proteins in vitro, and used Pareto-frontier characterization to yield interpretable scientific insights.The work also introduced a large-molecule benchmark for evaluating biological sequence-design methods.Source: §6 · p. 9
• Combining LaMBO with pretrained antibody generative models and improving its mutation-site initialization beyond uniform random sampling are proposed as short-term directions.The authors identify both as opportunities for future work, not as evaluated improvements.Source: §6 · p. 9
• Further sequence-design directions include non-myopic and multi-fidelity acquisition, explicit design constraints, portfolio-level risk measures, multimodal inputs, and greater resilience to model misspecification, miscalibration, and covariate shift.The authors also highlight the need for better drug-design benchmarks.Source: §6 · p. 9
• The authors describe Bayesian optimization as having a path to affect drug design and scientific discovery, while acknowledging that many challenges remain.This is framed as a prospective possibility rather than a demonstrated outcome.Source: §6 · p. 9

Appendices

• The appendices cover four evaluation tasks and provide architecture, GP, autoencoder, string-kernel, hyperparameter, and supplementary-results details.The listed tasks are Bigrams, logP + QED, DRD3 docking + SA, and Stability + SASA.Source: Appendices · p. 14

A.1. Bigrams Task

• The Bigrams task maximizes counts of three predetermined bigrams in random-alphabet strings of up to 36 tokens, including utility tokens.The task is presented as a simple toy example of discrete sequence optimization.Source: §A.1 · p. 14

A.2. logP + QED Task

• The logP + QED task jointly optimizes hydrophobicity and a drug-likeness metric because logP alone is flawed and QED also has limitations.The authors note that logP can reward long hydrocarbon chains, while QED is a small-data parametric model of heuristics; the start pool uses dominated ZINC sequences to leave room for improvement.Source: §A.2 · p. 14
• Existing large-molecule in-silico evaluations lacked rigorous tasks, motivating the proposed RFP benchmark; existing molecule-generation benchmarks were limited to ZINC-like QED and logP optimization.The authors also note that prior large-molecule tasks were relatively simple, single-objective problems.Source: §A.2 · p. 14

A.3. DRD3 Docking + SA Task

• The DRD3 docking + SA task pairs docking affinity evaluation with synthetic accessibility because docking-only optimization may yield molecules that are difficult to synthesize.The task uses the DRD3 docking score oracle and a shared start pool of 512 randomly selected ZINC molecules labeled with the objective oracles.Source: §A.3 · p. 15

A.4. Stability + SASA Task

• The benchmark simulates in-vitro search for red fluorescent protein variants by optimizing stability and solvent-accessible surface area using FoldX and BioPython.The authors connect stability to protein function and SASA to factors related to brightness and photostability; both objectives depend on three-dimensional structure.Source: §A.4 · p. 15
• The starting set draws red-spectrum proteins with known structures and at most 244 residues from FPBase, then adds labelled sequences collected with NSGA-2.Red-spectrum proteins are defined here as having emission wavelength of at least 580 nm; the highest-resolution structure and longest chain are selected when needed.Source: §A.4 · p. 15
• Because FoldX computes structures only from substitution mutations, the benchmark cannot vary sequence length.The authors mention deep-learning structure oracles as alternatives but chose FoldX as simpler and more suitable for rapid prototyping.Source: §A.4 · p. 15

A.5. Wet lab experimental procedure

• Fluorescent proteins were synthesized in PUREfrex reactions from purchased DNA fragments, with fluorescence peaks measured by excitation and emission sweeps and thermostability measured as Tm using NanoDSF.Reactions ran overnight at 30°C; Tm is the midpoint transition temperature of the thermal melt curve.Source: §A.5 · p. 15

B. Implementation Details

• The models use PyTorch, BoTorch, and GPyTorch, while genetic optimizer baselines use PyMOO; code and hyperparameters are referenced.The code is publicly available at the cited project repository.Source: B. Implementation Details · p. 15

B.1. Architecture Details

• The shared encoder and decoder use convolutional residual blocks, while a single residual block forms the discriminative encoder; the multi-task GP uses ICM kernels.The architecture uses 1D convolutions with kernel size 5, 64 intermediate channels, and 16 latent channels; the authors chose small convolutions for speed and adequate performance.Source: §B.1 · p. 16

B.2. DKL Implementation Details

• For DKL GP training, the authors recommend constraining kernel lengthscale adaptation, reducing Adam moment estimates, and using layer normalization rather than batch normalization.A tight Gaussian prior with σ = 0.01 is placed around the initial lengthscale value.Source: Applicable to exact and variational GP… · p. 16

Applicable to variational GP inference

• For variational GP inference, retraining reinitializes inducing points and variational parameters, and alternating epochs can disable GP-parameter gradients.The reinitialization procedure is reported to improve results and save computation.Source: Applicable to variational GP inference · p. 16
• With suitable procedures, DKL SVGPs can reach accuracy similar to exact DKL GPs; joint training takes one supervised GP-loss step and one unsupervised DAE-loss step per minibatch.The models use diagonal Gaussian likelihoods, and the authors reserve 10% of collected data for validation and early stopping because DKL GPs can overfit.Source: Applicable to variational GP inference · p. 16

• The MLM head is trained by masking 0.125 of input tokens and minimizing cross-entropy between original tokens and predictions at masked positions.Source: §B.3 · p. 16
• During sequence optimization, the MLM predictive distribution excludes the original token and special tokens to encourage diversity.Source: §B.3 · p. 17
• The LANMT head adds length prediction and a length transform; training predicts both the original length and the full decoded sequence.For Figure 10, corruption randomly changed length by -8 to 8 tokens through deletion, replacement, or insertion.Source: §B.3 · p. 17

• Across offline regression, DKL MTGPs and SVGPs perform well broadly, while bootstrapped CNN ensembles are overconfident and substring-kernel GPs are underconfident and inaccurate on logP and QED.Source: Figure 9 · p. 18
• On RFP SASA prediction, the SSK GP was under-confident and less accurate than the deep-kernel GP in offline regression.The comparison used 410 training examples and 102 validation and test examples, and did not evaluate the models online.Source: §C.1 · p. 18, Figure 8 · p. 18
• Substring kernels lack positional awareness: matching n-grams can raise covariance regardless of their sequence locations, obscuring function-relevant positions in related proteins.Source: §C.1 · p. 18
• SSKs are difficult to combine with deep generative models and standard GP scaling methods; dynamic programming becomes memory-prohibitive beyond 100-token sequences.A trie-based implementation handled longer sequences but was prohibitively slow and difficult to parallelize.Source: §C.1 · pp. 18–19

C.2. Comparison to LSBO for single-objective BBO

• For penalized logP, LaMBO used 2,000 high-scoring ZINC sequences and 8,000 random sequences as its start pool, with a length-changing LANMT decoder that allowed insertions.The decoder choice addressed the observation that high-scoring molecules can be larger than any in ZINC.Source: §C.2 · p. 19
• LaMBO exceeded LSBO's best reported penalized logP score of 27.84, reaching 50 for some seeds, but LSBO was more sample-efficient at 500 evaluations.LaMBO required about twice as many online observations to reach LSBO's reported median best score; the methods' sample efficiency is not directly comparable because LSBO used the entire ZINC dataset for pretraining.Source: §C.2 · p. 19, Figure 10 · p. 19

C.3. Ablating LaMBO: Proposal Entropy and Discriminative Performance

• Adding the unsupervised DAE objective to supervised GP training improved discriminative performance, while the ablation methods otherwise behaved similarly, implicating better proposals in the better solutions.The entropy penalty reduced proposal entropy.Source: Figure 11 · p. 20