PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating and designing DNA with deep generative models

Nathan Killoran, Leo J. Lee, Andrew Delong, David Duvenaud, Brendan J. Frey

arXiv:1712.06148v1cs.LGq-bio.GNstat.ML

TL;DR

The paper addresses how to generate DNA sequences and tune them to desired properties while capturing structure in genomic data. It introduces a GAN-based generator, DNA activation maximization, and a joint generator–predictor procedure, and reports that these tools can design protein-binding probe sequences estimated to exceed training-data properties. The work presents generative DNA design as an initial, computationally validated research direction requiring further experimental validation.

  • Problem

    Existing generative tools had not been extensively explored for generating DNA sequences and tuning them to desired genomic properties.

  • Method

    The paper combines a GAN-based DNA generator, a DNA variant of activation maximization, and a joint architecture that tunes generated sequences through predictors.

  • Results

    The methods capture important DNA structure and generate protein-binding probe sequences with estimated binding values exceeding anything in the training data, in some cases by three times.

  • Takeaways & Limitations

    These tools provide an initial framework for exploring and designing new DNA sequences with desired properties.

  • Takeaways & Limitations

    Designed sequences require further improvement and wet-lab validation because cellular machinery ultimately determines their quality and utility, especially in critical applications.

Abstract

from arXiv · show

We propose generative neural network methods to generate DNA sequences and tune them to have desired properties. We present three approaches: creating synthetic DNA sequences using a generative adversarial network; a DNA-based variant of the activation maximization ("deep dream") design method; and a joint procedure which combines these two approaches together. We show that these tools capture important structures of the data and, when applied to designing probes for protein binding microarrays, allow us to generate new sequences whose properties are estimated to be superior to those found in the training data. We believe that these results open the door for applying deep generative models to advance genomics research.

1 Introduction

The paper applies deep generative methods to DNA, treating sequences as structured data with both language-like discrete organization and vision-like motifs. It presents complementary tools for generating sequences, tuning their properties, and exploring genomic design applications.

  • Deep generative methods can synthesize data, explore configuration spaces, and tune generated examples toward desired properties.The introduction also highlights potential applications to molecular design and genomics.
  • DNA combines discrete nucleotide sequences and hierarchical structure with recurring motifs on potentially noisy backgrounds.The paper therefore adapts design patterns from natural language and computer vision.
  • Genomic applications include designing protein-binding probes and optimizing sequences under multiple, possibly competing, properties.Examples include maintaining GC content while specifying translation or maximizing valuable chemical production.
  • The paper presents a GAN-based generator, a DNA variant of activation maximization, and a joint method combining generation with property tuning.Computational experiments examine whether these methods capture dataset structure and tune DNA sequences toward specific properties.

2 Generative Design of DNA

The paper develops generative modelling and optimization tools for DNA by combining a WGAN-based generator with predictor-guided sequence design. Its joint approach preserves learned data structure while tuning sequences toward desired properties in a continuous latent space.

  • Generative modelling: Generative models learn a structured subset of plausible data configurations rather than arbitrary possible sequences.Training adjusts the generator so its samples approximate examples from the real-data distribution.
  • Generative modelling: The paper uses a GAN-based generative model, after considering recurrent networks, PixelCNN, VAEs, and GANs for genomic sequence generation.The selected WGAN uses a generator and discriminator trained in alternating phases.
  • Generative optimization: In activation maximization, gradients modify an input toward a desired property, while a predictor may be explicit, learned, or a weighted combination of both.The predictor computes a target property from the input sequence.
  • Generative optimization: DNA activation maximization uses a continuous latent relaxation of one-hot sequences, applies gradients to that latent representation, and takes position-wise argmax values afterward.The softmax pre-layer provides normalized nucleotide distributions at each sequence position.
  • Joint method: The joint method combines a generator that captures generic structure with predictors that fine-tune specific properties, allowing independently trained components to be mixed and matched.This addresses the tendency of unconstrained activation maximization to ignore realism or typicality.

3 Computational Experiments

Computational experiments show that the DNA generator captures latent sequence structure, exon splice-site signals, and motif or protein-binding properties, enabling sequence design beyond the training data.

  • The WGAN was evaluated through latent-space exploration, sequence-distance analysis, exon modeling, and DNA-binding design experiments.These experiments tested both learned structure and the ability to optimize generated sequences for specified properties.
  • 3.1.1 Exploring the Latent Encoding: Latent interpolation produced approximately one-hot nucleotide outputs that changed smoothly between decoded sequences.Most latent points selected one dominant nucleotide channel, while transitions between sequences were gradual.
  • 3.1.1 Exploring the Latent Encoding: Reflection in latent space generated sequences strongly biased toward complementary bases, including C as the complement of G.The model appeared to encode DNA complementation through latent-space negation and showed similar structure for other base pairs.
  • 3.1.1 Exploring the Latent Encoding: Generated sequences were, on average, farther from training sequences than held-out test sequences, indicating that the GAN had not overfit.Both generated and test sequences remained concentrated around similar edit distances to the training set.
  • 3.1.2 Capturing Exon Splice Site Signals: The GAN-generated exon sequences approximated splice-site signals observed in the training data.The model jointly learned exon locations through an annotation track and nucleotide statistics near exon borders.
  • 3.2 DNA Sequence Design: Joint optimization generated sequences containing desired motifs and protein-binding scores above anything seen in the restricted training set.After removing scores above the 40th percentile, some designed sequences reached binding values three times higher than anything in the training data; the method also explored subtle differences between related binding objectives.

4 Summary & Future Work

The paper introduces deep generative tools for learning structure in DNA and designing sequences with desired properties, while emphasizing that practical use requires further validation and refinement.

  • The framework combines GAN-based DNA generation, DNA-specific activation maximization, and a joint method integrating both approaches.
  • The experiments indicate that these tools learn important DNA structure, even from restricted information, and can explore sequences with desired properties.
  • Wet-lab experiments are needed to improve and fine-tune the tools before real-world use, especially where single-nucleotide changes can have dramatic biological consequences.The authors state that cellular machinery ultimately determines the quality and utility of designed sequences.
  • The joint design framework can incorporate additional objective terms to penalize known undesirable sequence behaviors, such as multiple strong motif matches.These penalties are intended to suppress known undesirable outcomes in generated sequences.
  • The authors describe the work as an initial exploration and proof of concept for generative DNA design, with architectures designed to accommodate future advances.They expect improvements in generative models and genomics datasets to enhance the frameworks.

A Further Experiments on the Latent Space

Further experiments show that latent-space reflections can encode DNA complementation, although this behavior is strongest for highly regular, low-entropy sequences.

  • The latent-space experiments used a WGAN trained on human chromosome 1 DNA sequences to examine learned complementation structure.
  • Reflecting latent vectors generated sequences biased toward the complementary nucleotide for each tested starting nucleotide.The observed pairings were A↔T and C↔G.
  • Across 24 nucleotide one-hot encodings, 22 trained models showed strong pairing between A and T through latent-space reflection.The two remaining models displayed the behavior after retraining with different random initialization and noise.
  • The retraining results suggest that complementation behavior is not dependent on the particular one-hot encoding, but may reflect the data distribution.The authors hypothesize that outlier models entered an alternative training parameter regime.
  • Complementation was primarily evident for sequences repeating the same character, while sequences without such regularity showed no obvious reflection correspondence.The authors interpret this as approximate complementarity encoded mainly for low-entropy sequences.

B.1 Models

The models use residual WGAN-GP generators and discriminators, convolutional DNA-binding predictors, and differentiable design choices that preserve gradient information during optimization.

  • Model operations are performed on batches, with learnable parameters consisting of weights, biases, and convolutional filters.
  • Every generator experiment uses a Wasserstein GAN with gradient penalty and residual architectures in both generator and discriminator.The models are based on a WGAN language-model architecture.
  • DNA-protein binding predictors use convolutional neural networks similar to DeepBind and pad sequences on both sides to allow partial matches.Padding assigns each nucleotide channel a value of 0.25.
  • Leaky ReLU activations replace standard ReLUs because they provide nonvanishing gradient information for almost every input value.The experiments use α = 0.1 in leaky_relu(x) = max(αx, x).
  • The joint framework can add a latent-space prior term, ϵ2∇z log p(z), to encourage designed sequences to remain closer to prior latent values.The coefficient ϵ2 is a user-specified hyperparameter.

B.2 Datasets

The study uses three human-genome-derived datasets covering chromosome sequences, exon-centered sequences, and experimentally measured protein-binding probes.

  • The chr1 dataset contains length-50 nucleotide sequences compiled from chromosome 1 of the human hg38 genome assembly.Sequences were uppercased and entries containing unknown nucleotides were processed out.
  • The exons-50-400-500 dataset contains human exon sequences 50–400 nucleotides long with flanking regions forming 500-nucleotide sequences.Each exon is positioned at the center of its sequence.
  • The protein-binding dataset contains 36-nucleotide probe sequences with experimentally measured binding values across four protein-family subsets.Some families include measurements for one protein at multiple concentrations.

C Generative Modeling Architectures

RNNs generate DNA sequences autoregressively by processing nucleotides sequentially, but their design flexibility is limited because desired properties generally require supervised conditional training.

  • Recurrent neural networks: RNNs process sequence inputs sequentially while retaining an internal state that summarizes prior information.At each time step, the model combines the current input with its previous state to produce an output and updated state.
  • Recurrent neural networks: RNN training predicts the next character from the current input and learned sequence history, beginning with a data-independent start character.Generation samples each prediction and feeds it back as input for the next time step.
  • Suitability for DNA design: RNN-based DNA design lacks a successful activation-maximization or plug-and-play method, leaving vanilla RNN generation as the currently viable design option.Conditional RNNs can tune properties when sufficient supervised data are available.
  • Suitability for DNA design: Without a learned latent encoding, RNNs can tune only properties explicitly represented in supervised training, while discovering other useful activations may be time-consuming and uncertain.Unsupervised inspection may identify neurons associated with sequence properties, but there is no guarantee it will find the properties of interest.

C.2 Deep autoregressive models

Deep autoregressive models condition each prediction on the full preceding sequence and can generate tailored sequences, but DNA design still requires predefined supervised properties.

  • Deep autoregressive models: Deep autoregressive models replace reliance on recurrent memory with a deep network that processes the entire available history.The processing network may be recurrent or convolutional, as in PixelRNN and PixelCNN architectures.
  • Architectural examples: Figure 12 compares recurrent neural networks, PixelCNN, variational autoencoders, and generative adversarial networks using short example sequences.RNNs and PixelCNN receive a generic starting character at the first time step.
  • Deep autoregressive models: These models generate sequences by feeding previous predictions back as input at each subsequent time step.They extend the autoregressive procedure used by recurrent networks.
  • Deep autoregressive models: Conditional deep autoregressive models can generate sequences with tailored properties.The conditioning mechanism allows generation to depend on specified properties.
  • Suitability for DNA design: For DNA design, these models still require supervised training for every property of interest, which must be selected and encoded before training.Their greater access to past context does not remove this design constraint.

C.3 Variational Autoencoders

VAEs learn controllable latent representations that can modify generated sequences, but sequence data—especially noisy genomic data—can cause the latent code to be ignored.

  • Latent representation: VAEs learn a latent representation that can be varied to modify generated data, providing potential flexibility for sequence tuning.Unlike the preceding autoregressive models, this representation can be learned without supervised property labels.
  • VAE architecture: A VAE encodes data into a latent variable through q(z|x) and decodes latent variables into generated data through p(x|z).Both transformations are modeled probabilistically by neural networks.
  • VAE objective: VAE training minimizes reconstruction error while regularizing q(z|x) toward a prior p(z) through a KL-divergence term.The reconstruction objective measures the error of the encoding-decoding path, while the KL term encourages smooth latent codes.
  • Suitability for DNA design: Strong decoders can drive the KL divergence toward zero, causing the decoder to ignore the latent code and undermining the intended representation.Proposed remedies include temporarily removing KL regularization or restricting decoder capacity.
  • Empirical behavior: VAE+RNN models were unreliable on realistic noisy genomics data, where near-zero KL divergence led them to model sequences through the decoder alone.This behavior persisted even when KL regularization was removed during early training.
  • Stochastic decoding: Because VAEs decode stochastically, one latent code may produce multiple samples, complicating control over how generated variation is allocated between the code and decoder noise.This stochasticity may interfere with fine-tuning outputs.

C.4 Generative Adversarial Networks

GANs use a deterministic generator whose latent code can support continuous sequence tuning, while training remains vulnerable to instability and mode collapse.

  • Generative adversarial networks: GANs provide a deterministic generator, so variation in generated sequences is attributed to the latent code rather than stochastic decoding.This contrasts with the stochastic generation process of VAEs.
  • Latent-space design: Moving through GAN latent space can continuously modify and tune generated sequences because the latent code serves as a high-level representation.The passage characterizes this representation as reliable relative to the latent-code behavior discussed for VAEs.
  • Training limitations: GAN training can become unstable when the generator and discriminator dominate one another or enter oscillatory optimization patterns.Such dynamics can produce suboptimal generative models.
  • Training limitations: GAN generators may concentrate samples on a small number of modes, ignoring variation present in the real data.The WGAN architecture used in this work has been effective against many of these instabilities.
Loading 1712.06148v1…