Deep learning in bioinformatics: introduction, application, and perspective in big data era

Yu Li; Chao Huang; Lizhong Ding; Zhongxiao Li; Yijie Pan; Xin Gao

Source-linked AI summary

Deep learning in bioinformatics: introduction, application, and perspective in big data era

Yu Li, Chao Huang, Lizhong Ding, Zhongxiao Li, Yijie Pan, Xin Gao

arXiv:1903.00342v1Feb 28, 2019q-bio.QMcs.LGcs.NE

TL;DR

Bioinformatics is generating enormous biological datasets for which deep learning is well suited, but adopting these methods requires attention to recurring difficulties such as overfitting and interpretability. This review explains major deep-learning architectures, demonstrates representative bioinformatics applications with implementations, and offers suggestions for common challenges.

Problem
The enormous amount of biological data creates a need for analysis methods suited to sequence, expression, network, and other bioinformatics data types.
Method
The review introduces deep-learning architectures, explains their components, and provides concrete bioinformatics examples with TensorFlow and Keras implementations.
Results
The review covers deep-learning achievements, architectures, applications, implementations, and common adoption difficulties in bioinformatics.
Takeaways & Limitations
Deep learning can support diverse bioinformatics analyses, while users should address model complexity, overfitting, interpretability, uncertainty, and catastrophic forgetting.
Takeaways & Limitations
High model complexity and many interrelated parameters create a risk of overfitting and poor generalization to testing data.

Abstract

from arXiv · show

Deep learning, which is especially formidable in handling big data, has achieved great success in various fields, including bioinformatics. With the advances of the big data era in biology, it is foreseeable that deep learning will become increasingly important in the field and will be incorporated in vast majorities of analysis pipelines. In this review, we provide both the exoteric introduction of deep learning, and concrete examples and implementations of its representative applications in bioinformatics. We start from the recent achievements of deep learning in the bioinformatics field, pointing out the problems which are suitable to use deep learning. After that, we introduce deep learning in an easy-to-understand fashion, from shallow neural networks to legendary convolutional neural networks, legendary recurrent neural networks, graph neural networks, generative adversarial networks, variational autoencoder, and the most recent state-of-the-art architectures. After that, we provide eight examples, covering five bioinformatics research directions and all the four kinds of data type, with the implementation written in Tensorflow and Keras. Finally, we discuss the common issues, such as overfitting and interpretability, that users will encounter when adopting deep learning methods and provide corresponding suggestions. The implementations are freely available at \url{https://github.com/lykaust15/Deep_learning_examples}.

1 Introduction

Deep learning has become increasingly useful in bioinformatics because biological data are abundant and often suitable for automated feature extraction. This review surveys applications, explains core architectures, and provides tutorial implementations across major bioinformatics directions and data types.

Applications: Deep learning has been applied to sequence analysis, structure prediction, biomolecular property and function prediction, biomedical imaging, and interaction prediction.Examples include motif detection, structure reconstruction, function prediction, diagnosis, drug-target interaction, and systems biology.
Motivation: Biological data abundance is a central reason deep learning is suitable for bioinformatics analysis.The review identifies sequence, image, graph, and preprocessed biological data as application domains.
Method suitability: Deep learning is most advantageous for raw data because it can perform end-to-end feature extraction and classification, whereas gains may be limited for already structured data.For structured data, the review states that carefully tuned deep learning need not outperform conventional methods such as SVM.
Review scope: The review provides a detailed introduction to neural-network architectures and concrete examples spanning five research directions and four biological data types.Covered architectures include fully connected networks, CNNs, RNNs, ResNets, GANs, VAEs, and GCNs.
Review scope: Unlike many prior reviews, the paper explains algorithms step by step and supplies implementations intended to help biologists adopt deep learning methods.The implementations are designed to bridge machine-learning developers and biology end users.

2 From shallow neural networks to deep learning

The review presents a progression from shallow neural networks to architectures designed for increasingly varied biological data and modeling tasks.

Architecture progression: The review introduces shallow neural networks, CNNs, RNNs, and newer architectures including ResNet, DenseNet, and SENet.It then extends coverage to graph neural networks, generative models, and other state-of-the-art architectures.
Architecture progression: Graph neural networks are presented for network data, while GANs and VAEs are introduced as important generative models.The sequence moves from regular one- and two-dimensional data to graph and generative modeling.

2.1 Shallow neural networks and their components

A shallow neural network composes linear transformations and nonlinear activations into a multilayer mapping trained by forward propagation, back-propagation, and optimization.

Network components: Each neural-network node applies a weighted linear transformation with bias followed by a nonlinear activation function.The resulting node output becomes part of the next layer’s input.
Network components: Aggregating node functions into layers and stacking layers enables the network to express complex nonlinear mappings.The model’s representational complexity increases as more building blocks are combined.
Training procedure: Training alternates forward propagation, loss calculation against targets, error back-propagation, and parameter optimization until convergence or a preset iteration limit.After training, inference uses forward propagation alone.
Design choices: ReLU is commonly used in hidden layers and Softmax in output layers, while node count must balance underfitting against overfitting.Too few nodes can make the model too simple, whereas too many can fit noise.

2.2 Legendary deep learning architectures: CNN and RNN

CNNs model spatial structure through local, shared convolutional filters and pooling, while RNNs process inputs sequentially to capture temporal or order dependencies.

Motivation: Shallow neural networks can have enormous parameter counts and ignore correlations between input features, causing overfitting and slower computation.With layers of N1 and N2 nodes, the connections between them contain N1 ∗N2 parameters.
Convolutional neural networks: CNNs apply convolution and nonlinear activation operations, typically followed by pooling, flattening, and fully connected layers.Convolutional operations may be repeated across channels before the final output is produced.
Convolutional neural networks: Local connectivity makes each output element depend on part of the input, while weight sharing lets filters detect recurring patterns or motifs.The same weight vector slides across different input regions during convolution.
Convolutional neural networks: Pooling selects regional features, enabling CNNs to capture higher-level and longer-range properties alongside local motifs.Max pooling takes the maximum within each corresponding input region.
Recurrent neural networks: RNNs feed sequential inputs element by element so hidden states can retain temporal or order information that independent-feature models may lose.This is relevant when motif order influences DNA-sequence function or when word order affects document categorization.

2.3 State-of-the-art deep architectures

CNN and RNN architectures have developed substantially beyond their standard forms, producing families of specialized architectures for different modeling needs.

Convolutional architectures: Advanced CNN architectures include AlexNet, VGG, GoogleNet, ResNet, ResNext, SENet, DenseNet, and DPN.These architectures represent successive developments for handling image data with convolutional networks.
Recurrent architectures: Advanced RNN architectures include LSTM, Bi-RNN, GRU, Memory network, and Attention network.These variants extend recurrent modeling beyond the basic RNN structure.

2.4 Graph neural networks

Graph neural networks address irregular network data by aggregating neighborhood information into embeddings that preserve topology and support prediction tasks.

Graph data: Bioinformatics network data are irregular, but their topology and interactions can carry biologically meaningful information for classification or prediction.Unlike sequences and images, network nodes may have arbitrary connections.
Graph convolution: Graph convolutional networks update each node by aggregating representations from neighboring nodes and applying trainable transformations and activation functions.The layer-wise process builds embeddings from node properties and network context.
Graph convolution: Shared parameters across nodes and neighbor trees allow the graph neural network to generalize to previously unseen graphs of the same type.The same-layer weights are reused between different nodes and trees.
Training objectives: Graph embeddings can be trained using similarity objectives or by stacking a classifier, CNN, or RNN on top of the graph network.The embedding objective may encourage similar nodes to receive similar representations.

2.5 Generative models: GAN and VAE

GANs and VAEs are generative models for learning data structure and producing new samples, using adversarial discrimination or probabilistic latent representations.

Generative models: Generative models focus on intrinsic data properties and can support biological image processing, protein design, and drug design.They differ from supervised models that perform classification or regression.
GAN: GANs train a generator and discriminator competitively so generated samples become increasingly difficult to distinguish from real data.For enzyme sequences, the discriminator guides the generator toward sequences more likely to be real enzyme sequences.
VAE: VAEs map inputs to distributions in latent space rather than fixed vectors, enabling sampling of new latent vectors for data generation.A decoder then generates data from each sampled latent vector.
Autoencoder: Autoencoders compress high-dimensional inputs into lower-dimensional latent vectors and reconstruct the inputs through a decoder.The reconstruction loss commonly measures the difference between the original input and decoder output.

2.6 Frameworks

The review identifies commonly used deep-learning frameworks that reduce the effort required to build and train models.

2.6 Frameworks: Frameworks such as TensorFlow, PyTorch, Caffe2, MXNet, CNTK, and PaddlePaddle accelerate network construction and model training.The authors describe these frameworks as actively developed and maintained.

3 Applications of deep learning in bioinformatics

The review presents eight representative deep-learning examples spanning major bioinformatics directions, model families, learning settings, and data types. These examples include enzyme identification, gene-expression regression, and RNA-binding-site prediction.

Coverage: The eight examples collectively cover the major bioinformatics research directions, deep-learning models, data types, and tasks discussed in the review.The examples are summarized in Table 4.
Applications: The examples include enzyme-sequence identification, gene-expression prediction, and RNA-binding-protein binding-site prediction.These tasks illustrate classification, regression, and sequence-pattern analysis applications.
Applications: 94.5% accuracy was reached for enzyme identification after two minutes of training on a Titan X.The reported accuracy was close to the state-of-the-art performance.
Applications: 4.5% improvement over linear regression was achieved for a randomly selected target gene in gene-expression prediction.The network was trained for two minutes on a Titan X using mean squared error for regression.
Applications: CNN-based RNA-binding-site prediction uses one-hot encoded RNA sequences to detect sequence patterns relevant to binding.The encoded sequence can be treated as a one-channel 2D map or a four-channel 1D vector, enabling corresponding convolution choices.

3.4 DNA sequence function prediction with CNN and RNN

This application uses CNNs and RNNs to predict the functionality of non-coding DNA sequences. It addresses the difficulty of experimentally characterizing the large non-coding portion of the human genome.

Motivation: Over 98% of the human genome is non-coding DNA, making comprehensive experimental functionality assessment difficult.The review motivates computational prediction as a cheaper alternative.
Method: CNN and RNN models predict non-coding DNA functionality from 1000-bp sequences centered on non-overlapping 200-bp genome bins.The data come from the human GRCh37 reference genome, with labels derived from ENCODE and Roadmap Epigenomics profiles.
Extensions: The same prediction setup can be adapted to DNA methylation-state prediction and long non-coding RNA function prediction.These applications are presented as similar sequence-prediction problems.

3.6 Graph embedding for novel protein interaction prediction using GCN

The review demonstrates graph embedding for predicting novel protein-protein interactions in a yeast network. A graph neural network learns protein representations from network topology and scores candidate interactions from paired embeddings.

Task: The application targets novel protein-protein interaction prediction in the yeast PPI network.Interaction knowledge can support protein-function prediction and drug design.
Method: The graph neural network learns node embeddings for proteins from the topology of the interaction network.Known interactions and non-interacting protein pairs provide supervised training targets.
Prediction: The inner product of two protein embeddings serves as the interaction score, with scores above a threshold producing predicted interactions.The model is optimized with cross-entropy until stable node embeddings are obtained.
Extensions: The same graph-based approach can extend to symptom-disease networks, gene co-expression networks, and cell-system hierarchies.The review presents these as other common bioinformatics graph problems.

4 Perspectives: limitations and suggestions

The review identifies recurring challenges in applying deep learning to bioinformatics, including data scarcity, overfitting, imbalance, interpretability, uncertainty calibration, continual learning, and computational cost. It pairs these challenges with strategies such as transfer learning, regularization, interpretation methods, probability scaling, and model compression.

4.1 Lack of data: Deep learning is data-hungry, and biological datasets may sometimes be insufficient for direct application.Suggested responses include transfer learning, fine-tuning pretrained models, data augmentation, and simulated data.
4.2 Overfitting: High model complexity creates substantial risks of overfitting and poor generalization to testing data.Dropout, batch normalization, and weight decay are among the proposed techniques for alleviating overfitting.
4.3 Imbalanced data: Biological datasets are often imbalanced, with positive samples greatly outnumbered by negative samples, potentially producing undesirable predictions.
4.4 Interpretability: Deep learning can be interpreted through example-specific importance scores that reveal influential input patterns, motifs, or evidences.Perturbation-based methods change input parts and observe output effects, whereas backpropagation-based methods propagate signals backward.
4.5 Uncertainty scaling: Softmax outputs may be overconfident, so post-scaling methods are needed to obtain reliable probability scores.The review lists Platt scaling, histogram binning, isotonic regression, and Bayesian Binning into Quantiles as calibration methods.
4.6 Catastrophic forgetting: Plain deep learning models may suffer catastrophic forgetting when new knowledge is incorporated, while retraining from old and new data is computationally intensive.This issue is especially relevant as biological databases continually accumulate new entries and classes.
4.7 Reducing computational requirement and model compression: Deep learning models are computationally demanding and memory intensive, limiting deployment on machines with restricted resources.The review discusses specialized hardware, distributed systems, cloud computing, and model compression as responses to these requirements.

5 Conclusion

The review surveys deep learning applications in bioinformatics, introduces major architectures, and provides concrete implementations to support adoption. It also discusses common application difficulties and corresponding suggestions, while acknowledging that some deep learning aspects remain outside its coverage.

5 Conclusion: The review introduces deep learning architectures from shallow and deep neural networks through convolutional, recurrent, graph, generative adversarial, and variational autoencoder models.
5 Conclusion: It provides detailed bioinformatics examples with implementations intended to help researchers adopt and develop deep learning methods.
5 Conclusion: The review identifies common difficulties in using deep learning and offers corresponding suggestions.
5 Conclusion: The review covers most aspects of deep learning applications in bioinformatics but does not cover all areas, including deep reinforcement learning and theoretical aspects.