Source-linked AI summary
Neural network models and deep learning - a primer for biologists
Nikolaus Kriegeskorte, Tal Golan
TL;DR
The paper addresses how neural-network models can inform understanding of brain computation beyond matching overall task performance. It introduces feedforward and recurrent networks, explains backpropagation, and concludes that deep networks provide a framework for rigorous tests of brain-computation theories while retaining important behavioral and neural incompatibilities.
Problem
Matching human or animal task performance does not establish that a neural network models the brain’s computations, motivating comparisons of detailed behavior and internal representations.
Method
The paper introduces feedforward and recurrent networks and explains backpropagation, gradient descent, and stochastic gradient descent for learning network parameters.
Results
Deep neural networks can meet or surpass human-level performance in some domains, while visual-recognition models remain the best available models of rapid primate recognition despite reported functional incompatibilities.
Takeaways & Limitations
Deep neural networks provide a framework for modeling brain information processing and subjecting theories of brain computation to rigorous simulation tests.
Takeaways & Limitations
Because the cost function is nonconvex, saddle points where the gradient vanishes pose a greater practical challenge for gradient descent than local minima.
Abstract
from arXiv · showhide
Originally inspired by neurobiology, deep neural network models have become a powerful tool of machine learning and artificial intelligence, where they are used to approximate functions and dynamics by learning from examples. Here we give a brief introduction to neural network models and deep learning for biologists. We introduce feedforward and recurrent networks and explain the expressive power of this modeling framework and the backpropagation algorithm for setting the parameters. Finally, we consider how deep neural networks might help us understand the brain's computations.
Neural network models of brain function
Artificial neural networks provide an intermediate, biologically inspired level for modeling brain information processing. By combining nonlinear units in hidden layers, they can represent nonlinear input–output functions and approximate any continuous function with sufficient units.
- Modeling brain function: Artificial neural networks occupy an intermediate modeling level between detailed single-neuron models and abstract, nonbiological information-processing algorithms.They were inspired by the idea that neurons categorize complex input patterns into simple decisions.
- Neural network units: A typical unit computes a weighted sum of inputs, adds a bias, and transforms the result through a nonlinear activation function.Threshold, logistic, and rectifying nonlinearities are examples of activation functions described in the passage.
- Expressive power: A hidden layer enables networks to discriminate classes that are not linearly separable in the input signals.Without nonlinear activation functions, stacked weighted sums remain limited to weighted sums of the inputs.
- Expressive power: Nonlinear hidden units can be combined by output units to produce continuous functions with shapes unlike the individual activation functions.Adjusting incoming weights changes the direction and scale of each hidden unit’s nonlinear ramp over input space.
- Universal approximation: A single hidden layer can approximate any continuous function to any desired precision when it contains a sufficient number of units.The construction can tile input space with compact patches formed by combining overlapping hidden-unit ramps.
Deep networks can efficiently capture complex functions
Deep networks exploit hierarchical feature reuse and compositional structure to approximate many natural functions with fewer parameters and greater precision. Deep learning determines parameters from data and supports supervised mappings as well as unsupervised representations and generation.
- Deep networks can efficiently capture complex functions: Deep networks reuse features across hidden layers, exploiting compositional structure to approximate many natural functions with fewer weights and units.Their hierarchical organization contrasts with shallow networks that piece functions together like a lookup table.
- Deep networks can efficiently capture complex functions: A deeper architecture can improve approximation precision under a fixed parameter budget and enhance generalization.
- Deep networks can efficiently capture complex functions: Deep learning automatically determines parameters deep in a network from experience, especially data.Although multiple-hidden-layer networks were studied in the 1980s, limited computing power and data initially constrained their practical potential.
- Deep networks can efficiently capture complex functions: Supervised training approximates mappings from inputs to desired outputs by minimizing output mismatch over example cases.Examples include image classification, English-to-French translation, and weather prediction.
- Deep networks can efficiently capture complex functions: Unsupervised learning includes autoencoders that compress inputs into latent representations and generative adversarial networks that produce synthetic category examples.
Deep learning by backpropagation
Backpropagation efficiently trains supervised deep neural networks by computing how the cost changes with each weight and adjusting weights to reduce error. The method uses forward activation propagation, derivative-based updates, and stochastic gradient descent, while saddle points pose a greater practical challenge than local minima.
- Learning weights: Supervised deep networks learn by randomly initializing connection weights and adjusting them in many small steps toward desired behavior.Random perturbation is presented as an intuitive alternative evolutionary approach.
- Learning weights: Differentiable operations allow partial derivatives of error with respect to all weights to be computed efficiently because their algebraic expressions share terms.This shared structure makes derivative computation more efficient than treating each weight independently.
- Backpropagation: Backpropagation first propagates activations forward, computes output cost against desired outputs, and then determines each unit’s sensitivity to activation changes.An example cost function is the sum of squared errors across output units.
- Backpropagation: A weight’s effect on cost depends on the activity of its source unit, so an inactive source transmits no signal and makes the connection weight irrelevant for that input.The derivative calculation traces the weight’s influence through the connected units.
- Gradient descent: Gradient descent updates each weight toward lower cost by an amount proportional to the cost derivative with respect to that weight.It is described as moving through weight space in the direction where cost declines most steeply.
- Gradient descent: Stochastic gradient descent uses batches as a compromise between gradient-estimate stability and computational cost, although nonlinearity prevents even exact full-training-set gradients from enabling arbitrarily large steps.The gradient estimate varies with the randomly sampled examples in each current batch.
- Optimization challenges: Because the cost is nonconvex, saddle points where the gradient vanishes are a greater practical challenge than local minima.High-dimensional weight space provides many directions for escaping configurations that are not true traps.