PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Deep Learning for Scientific Discovery

Maithra Raghu, Eric Schmidt

arXiv:2003.11755v1cs.LGstat.ML

TL;DR

Scientific datasets are growing in size and complexity, while the breadth of deep learning methods makes it difficult to choose an effective starting point. This survey synthesizes models, tasks, training methods, data-efficient and interpretable approaches, and implementation resources. It presents these tools as a way to support predictive problems and analysis of scientific data, while noting that biases and limited labels remain important constraints.

  • Problem

    The growing diversity of deep learning techniques makes it difficult to determine which methods are appropriate for scientific problems and how to begin implementing them.

  • Method

    The survey synthesizes deep learning models, tasks, training methods, data-efficient learning, interpretability, representation analysis, workflows, and implementation resources.

  • Results

    The survey provides a broad overview of successful deep learning models, tasks, and methodologies, together with community-developed open-source resources for scientific applications.

  • Takeaways & Limitations

    The survey offers guidance for applying deep learning across scientific domains, including settings with limited labels and needs for model interpretation.

  • Takeaways & Limitations

    Deep neural networks can learn biases in data, creating challenges for conclusions that must generalize across populations.

Abstract

from arXiv · show

Over the past few years, we have seen fundamental breakthroughs in core problems in machine learning, largely driven by advances in deep neural networks. At the same time, the amount of data collected in a wide array of scientific domains is dramatically increasing in both size and complexity. Taken together, this suggests many exciting opportunities for deep learning applications in scientific settings. But a significant challenge to this is simply knowing where to start. The sheer breadth and diversity of different deep learning techniques makes it difficult to determine what scientific problems might be most amenable to these methods, or which specific combination of methods might offer the most promising first approach. In this survey, we focus on addressing this central issue, providing an overview of many widely used deep learning models, spanning visual, sequential and graph structured data, associated tasks and different training methods, along with techniques to use deep learning with less data and better interpret these complex models --- two central considerations for many scientific use cases. We also include overviews of the full design process, implementation tips, and links to a plethora of tutorials, research summaries and open-sourced deep learning pipelines and pretrained models, developed by the community. We hope that this survey will help accelerate the use of deep learning across different scientific domains.

1 Introduction

Deep learning offers broad potential for scientific problems, but the diversity of models, tasks, and training methods makes it difficult to know where to begin. This survey addresses that challenge with an accessible overview, scientific-use techniques, and implementation resources.

  • The survey covers deep learning models for visual, graph-structured, and sequential data, along with diverse tasks and training methods.
  • Its explanations remain high level and concise so that core ideas are accessible to a broad audience and the survey can be read end to end.
  • For scientific applications, it details methods for using less data and techniques for interpretability and representation analysis.
  • The survey supports implementation through an overview of the design process, implementation tips, open-source code, research summaries, and tutorial references.
  • It is intended for readers with basic machine-learning knowledge who want a comprehensive overview and guidance for ramping up implementation.
  • The survey is organized around scientific applications, community resources, standard models and tasks, training variants, data efficiency, interpretability, advanced methods, and implementation tips.

2 High Level Considerations for Deep Learning

The survey frames scientific deep learning around prediction, understanding, and complex data transformations, then presents an iterative workflow for choosing data, learning, and validation procedures. It also emphasizes that deep learning should be compared with simpler or alternative methods when appropriate.

  • Templates for Deep Learning in Scientific Settings: Scientific applications of deep learning include prediction, understanding model-relevant mechanisms, and complex transformations of rapidly growing scientific data.
  • Templates for Deep Learning in Scientific Settings: Deep learning can automate complex scientific processing, including cell-nuclei segmentation in images and behavior labeling in videos.
  • Deep Learning Workflow: A typical system design has data, learning, and validation stages, but results from later stages often require earlier choices to be revisited.
  • Deep Learning Workflow: The survey focuses primarily on learning-stage models, tasks, and methods, while also covering data augmentation, interpretability, and representation analysis.
  • Deep Learning or Not?: Deep learning may not be the best starting point for every problem; dimensionality reduction and clustering can help reveal hidden data properties during exploration.
  • Deep Learning or Not?: Dimensionality-reduction methods include linear approaches such as PCA and non-linear approaches such as t-SNE and UMAP.

3 Deep Learning Libraries and Resources

The survey highlights a broad ecosystem of tutorials, research overviews, open-source code, pretrained models, and software for implementing deep learning systems. These resources support data preparation, model development, training, visualization, and implementation.

  • Community resources: The deep learning community provides tutorials, research overviews, open-source code, and pretrained models across many topics.These resources include frameworks, implementation repositories, and model collections.
  • Software libraries: PyTorch and TensorFlow provide core tensor operations and automatic differentiation for developing and training neural networks.PyTorch also offers Lightning, while TensorFlow offers Keras as a high-level API.
  • Implementations and models: Public repositories provide implementations of multilayer perceptrons, convolutional networks, GANs, recurrent networks, reinforcement-learning algorithms, and graph neural networks.Other resources include image-classification architectures, model links, and benchmarking code.
  • Data preparation: Data collection, curation, and labeling are crucial, time-intensive, and often intricate steps in applying deep learning.Image-labeling tools support tasks such as object-boundary annotation for segmentation.
  • Visualization and compute: TensorBoard helps visualize training metrics, while Colab provides an interactive development environment and free compute resources.The survey presents these tools as resources for model analysis and development.

4 Standard Neural Network Models and Tasks

The survey organizes standard neural network models and learning tasks by data modality and prediction objective. It introduces supervised learning, multilayer perceptrons, and scientific applications of simple MLPs.

  • Overview: The survey covers neural network models and tasks spanning convolutional, transformer, and graph-based approaches for different data modalities.The overview includes visual, sequential, and graph-structured data.
  • Supervised learning: Supervised learning trains neural networks from data instances paired with associated labels.The model parameters are selected so inputs produce corresponding labels; supervised learning can also reformulate settings without clear pairs.
  • Multilayer perceptrons: Multilayer perceptrons are stacks of fully connected layers with nonlinear transforms between matrix multiplications.They are suited to sets of distinct, potentially categorical features and offer more expressive power than linear or logistic regression.
  • Scientific examples: Simple MLPs have been used for pharmaceutical formulation by developing drug variants that are stable and safe for patient use.

4.3 Convolutional Neural Networks

Convolutional neural networks support image prediction and transformation tasks by reusing parameters across spatial locations. The survey describes classification, detection, segmentation, super-resolution, registration, and pose estimation, including scientific applications.

  • CNN foundations: Convolutional layers reuse parameters across image locations, providing an inductive bias that helps learn image features efficiently.The survey notes that some learned features correspond to traditional computer-vision techniques such as Gabor filters.
  • Classification and detection: Image classification assigns a single class to an entire image, whereas object detection identifies objects and localizes each with a bounding box.Detection architectures commonly combine an image-classification backbone with a region-proposal component.
  • Segmentation: Semantic segmentation labels pixels by group, while instance segmentation additionally distinguishes individual objects within each group.Segmentation architectures preserve spatial information because their outputs are full images; U-net is a popular scientific architecture.
  • Scientific applications: Segmentation methods have been applied to brain MR images, tissue-cell regions, and bone structure in biomedical research.
  • Image transformation: Super-resolution transforms low-resolution images into high-resolution images, an underdetermined problem with potentially many valid mappings.Scientific examples include fluorescence microscopy, electron microscopy, chest CT, and brain MRI.
  • Image registration: Image registration aligns two images, including images from different modalities or a moving image and a canonical template, to support feature identification and analysis.

4.4 Graph Neural Networks

Graph neural networks address datasets whose entities and relationships form graphs, supporting predictions at the vertex or whole-graph level. They are especially prominent in chemistry applications.

  • Graph data: Graph neural networks operate on graphs whose nodes and sometimes edges have associated feature vectors.Examples include atoms as nodes and chemical bonds as edges.
  • Prediction tasks: Graph-learning problems may predict properties for individual vertices, global properties of an entire graph, or properties of graphs evolving over time.The survey identifies spatio-temporal graph neural networks as a related variant.
  • Architectural pattern: Many graph neural networks aggregate information from neighboring vertices and use it to update feature vectors.Architectural variants connect graph networks with convolutional and recurrent models.
  • Scientific applications: Graph neural networks have been used in chemistry to predict molecular properties, determine protein interfaces, and generate candidate molecules.DeepChem is identified as a useful library for several of these chemistry tasks.

4.5 Neural Networks for Sequence Data

Sequence data appears across language, biological sequences, and other variable-length settings, motivating specialized tasks and models. The section surveys language modeling, sequence-to-sequence prediction, recurrent networks, attention, and Transformers, including their scientific applications and trade-offs.

  • Sequence data includes video frames, protein amino acids, and words, with tasks such as next-token prediction, sequence transformation, and similarity assessment.
  • Sequence tasks: Language modeling predicts token t + 1 from preceding tokens, providing self-supervised labels without additional annotation.
  • Sequence tasks: Sequence-to-sequence models transform one sequence into another, as in translating an English sentence into German with an encoder-decoder structure.
  • Sequence models: RNNs preserve and update an internal state across timesteps through feedback connections, but recurrent processing creates optimization and training-speed challenges.
  • Scientific applications: Sequence models have scientific applications in genomics and proteomics, while pretrained Transformers can be adapted to target problems.
  • Sequence models: Attention provides shortcut connections to earlier tokens, helping address long-range dependencies that are difficult for token-by-token RNNs to retain.
  • Sequence models: Transformers replace recurrence with feedforward processing of the entire sequence using self-attention and positional embeddings, becoming dominant across many language and sequential tasks.

4.6 Section Summary

This section summarizes the surveyed neural-network models and tasks while noting that several important structured-data and invariance topics remain outside its scope.

  • The survey has covered supervised learning, core neural-network models, and important tasks, but not every area of deep learning research.
  • Examples of omitted topics include set-structured data and invariances for molecular property prediction and spherical applications.
  • Transfer learning uses pretraining followed by fine-tuning, allowing learned features to be reused for a target task.

5 Key (Supervised Learning) Methods

The section surveys supervised-learning methods for training and adapting deep neural networks, including transfer learning, domain adaptation, multitask learning, and weak supervision. These methods address initialization, domain shift, multiple objectives, and limited high-quality labels, while requiring attention to training design and scope.

  • Supervised learning: Supervised learning trains neural networks to output labels from input examples, using training, validation, and test splits for optimization and evaluation.
  • Transfer learning: Transfer learning pretrains a model on a generic task and fine-tunes it on the target task, reusing learned features.
  • Transfer learning: Transfer learning can improve target-task performance and convergence speed while reducing the amount of labeled data required.
  • Transfer learning: Transfer learning is useful across computer vision and natural-language settings, but its precise effects remain incompletely understood.
  • Domain adaptation: Domain adaptation seeks models that perform on a labeled source domain while learning features invariant to differences in a target domain.
  • Multitask learning: Multitask learning uses multiple output heads with shared features, giving common parameters feedback from several predictive tasks.
  • Weak supervision: Weak supervision trains models with noisy or indirect labels when high-quality target-task labels and suitable pretrained models are unavailable.
  • Scope: The surveyed supervised-learning methods represent only a sample of the broader methodological landscape, which also includes uncovered areas such as multimodal learning.

6 Doing More with Less Data

This section surveys methods for reducing deep learning’s dependence on expensive labelled data. Self-supervised and semi-supervised learning exploit unlabelled examples, while augmentation and denoising further support data-efficient training.

  • Large, reliable labelled datasets can be prohibitively expensive in specialized domains such as medicine.
  • Self-supervised learning: Self-supervision automatically derives labels from raw data through pretext tasks, enabling unlabelled data to train neural networks.Examples include predicting image rotations and learning from sequential context.
  • Data augmentation and denoising complement self-supervised and semi-supervised learning as ways to do more with less data.
  • Self-supervised learning: Self-supervised learning can produce meaningful representations for analysis or downstream tasks, including transfer learning followed by task-specific fine-tuning.
  • Semi-supervised learning: Semi-supervised learning targets settings with limited labelled data and more unlabelled data, combining both sources through methods such as self-training and shared objectives.
  • Semi-supervised learning: Sharing features between labelled and unlabelled objectives leads to better representations, and semi-supervised learning can significantly boost model efficacy.

7 Interpretability, Model Inspection and Representation Analysis

Scientific applications often seek understanding of mechanisms rather than prediction alone, motivating interpretability methods that attribute individual predictions and inspect learned representations. The survey covers feature attribution, model inspection, and representational analysis techniques.

  • Scientific deep learning often aims to understand mechanisms underlying observed patterns, not merely produce accurate predictions.
  • Interpretability methods range from rough feature attributions to analyses of what model components and hidden representations have learned.
  • Feature attribution: Saliency maps use output gradients with respect to inputs to highlight regions important for a prediction, with variants such as SmoothGrad improving robustness.
  • Feature attribution: Feature ablation and perturbation methods estimate importance by measuring predictive changes, including Shapely values averaged over subsets containing a feature.
  • Model inspection: Model inspection methods probe hidden neurons or optimize inputs to reveal the features that activate internal network components.
  • Representation analysis: Representational analysis can compare hidden representations across languages and reveal grouping according to language families.

8 Advanced Deep Learning Methods

The survey introduces generative models and reinforcement learning as advanced methods that may require particular problem properties. Generative models learn distributions or latent encodings, while reinforcement learning learns action policies through environmental interaction.

  • Advanced methods can be intricate to implement and may require specific problem properties, such as an effective environment model or simulator for reinforcement learning.
  • Generative models: Generative models aim to model and sample high-dimensional data distributions or learn lower-dimensional latent encodings capturing important variation.
  • Generative models: StyleGAN2 produces high-quality human-face samples, although fully modelling the diversity of complex distributions remains challenging.
  • Generative models: Generative models include GANs, variational autoencoders, autoregressive models, and flow models, each using different generation mechanisms.
  • Generative adversarial networks: GANs pit a generator against a discriminator: the generator synthesizes data-like samples, while the discriminator distinguishes real from generated samples.
  • Flow models: Flow models generate data through invertible transformations, enabling exact likelihood computation.
  • Reinforcement learning: Reinforcement learning trains an agent to learn action sequences through interaction with an environment and feedback from rewards.

9 Implementation Tips

The survey recommends a staged implementation process: understand the data, establish simple and well-tested baselines, then diagnose performance and validate generalization carefully. It emphasizes that scientific applications require checking distribution shift, spurious correlations, and bias rather than relying on validation performance alone.

  • Explore Your Data: Explore dimensionality reduction, clustering, label reliability, class balance, and representation of subpopulations before model training.
  • Try Simple Methods: Start new problems with simple models, majority baselines, smaller datasets, and overfitting checks before scaling experiments.
  • Use Tested Choices: Use well-tested models, tasks, pretrained systems, and repositories to avoid debugging the dataset, objective, architecture, and optimization simultaneously.
  • Debugging Poor Performance: Diagnose poor validation performance by checking overfitting, optimization on small subsets, simpler tasks, noisy data, distribution shift, and error patterns.
  • Hyperparameters: Prioritize learning rate and its schedule when tuning, and use cross-validation to select hyperparameters before retraining on all training data.
  • Validate Thoroughly: Evaluate models on separately collected data because neural networks can exploit spurious correlations, such as hospital-specific imaging artifacts, instead of disease features.
  • Validate Thoroughly: Analyze representations for bias because correlations such as gender and age can undermine conclusions across populations; debiasing is an active research area.

10 Conclusion

The survey synthesizes deep learning models, tasks, methodologies, and community resources for increasingly large and complex scientific datasets. It aims to accelerate applications that support prediction and reveal properties of data-generation processes.

  • Growing scientific datasets create opportunities for deep learning in predictive problems and in revealing subtle properties of data-generation processes.
  • The survey overviews successful models, tasks, methodologies, and comprehensive open-source resources developed by the community.
  • The authors hope these overviews and references accelerate deep learning applications across varied scientific problems.
Loading 2003.11755v1…