PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards an integration of deep learning and neuroscience

Adam Marblestone, Greg Wayne, Konrad Kording

arXiv:1606.03813v1q-bio.NC

TL;DR

The paper addresses how neuroscience’s detailed account of neural mechanisms might be connected to machine learning’s emphasis on optimization. It proposes that brains combine biologically structured architectures with diverse, internally generated cost functions, and argues that candidate learning mechanisms can support efficient, targeted adaptation while leaving important biological and temporal limitations unresolved.

  • Problem

    Neuroscience and machine learning emphasize different explanatory perspectives, while the brain’s cost functions, specialized structures, and learning mechanisms remain incompletely connected to optimization-based accounts.

  • Method

    The paper formulates hypotheses about brain optimization, surveys biologically plausible learning mechanisms, and maps specialized computational problems onto brain structures.

  • Results

    The authors argue that biologically plausible neural networks can optimize complex functions of temporal history, using diverse mechanisms within specialized structures.

  • Takeaways & Limitations

    Brains may achieve data-efficient, precisely targeted learning through interacting cost functions that are diverse, developmentally regulated, and matched to specialized architectures.

Abstract

from arXiv · show

Neuroscience has focused on the detailed implementation of computation, studying neural codes, dynamics and circuits. In machine learning, however, artificial neural networks tend to eschew precisely designed codes, dynamics or circuits in favor of brute force optimization of a cost function, often using simple and relatively uniform initial architectures. Two recent developments have emerged within machine learning that create an opportunity to connect these seemingly divergent perspectives. First, structured architectures are used, including dedicated systems for attention, recursion and various forms of short- and long-term memory storage. Second, cost functions and training procedures have become more complex and are varied across layers and over time. Here we think about the brain in terms of these ideas. We hypothesize that (1) the brain optimizes cost functions, (2) these cost functions are diverse and differ across brain locations and over development, and (3) optimization operates within a pre-structured architecture matched to the computational problems posed by behavior. Such a heterogeneously optimized system, enabled by a series of interacting cost functions, serves to make learning data-efficient and precisely targeted to the needs of the organism. We suggest directions by which neuroscience could seek to refine and test these hypotheses.

1. Introduction

The paper proposes renewed convergence between neuroscience and machine learning by viewing the brain as a structured system with diverse, interacting cost functions. This framework aims to explain data-efficient learning through specialized architectures and internally bootstrapped objectives.

  • Machine learning has emphasized function optimization, whereas neuroscience has catalogued diverse mechanisms for computation and information storage.
  • Recent machine-learning systems use cost functions that vary across layers, time, and network interactions, improving feature learning and generalization.
  • Machine learning has diversified architectures with memory cells, specialized units, addressable memories, pointers, and hard-coded operations.
  • The paper frames these developments as hypotheses about the brain and proposes experiments to examine and refine them.
  • Brain cost functions may be locally generated or supplied by other areas, differ across locations, and change over development.
  • Specialized brain structures appear matched to distinct computational problems, while interacting developmental cost functions can bootstrap complex behavior from limited supervised information.

2. The brain can optimize cost functions

The paper argues that the brain is an optimization machine, using multiple forms of learning and credit assignment within genetically and developmentally structured systems. It proposes that specialized mechanisms and internally generated objectives support efficient adaptation across computational problems.

  • The brain may optimize global functions through credit assignment while using tunable cost functions shaped by evolution and matched to ethological needs.
  • The paper distinguishes local and multi-layer optimization and considers genetic circuit specification, local optimization, and other mechanisms for distinct problems.
  • Brain-like networks are presented as larger architectures containing specialized systems, with cortical areas schematized as trainable networks.
  • The proposed mechanisms are intended to support learning capabilities beyond those of current backpropagation networks.

2.1 Local self-organization and optimization without multi-layer credit assignment

The paper reviews learning mechanisms that can organize or optimize neural systems without full multilayer backpropagation. These include self-organizing and Hebbian processes, recurrent dynamical reservoirs, and readout-only training.

  • Self-organizing and unsupervised learning, including Hebbian plasticity, may support learning without multilayer backpropagation.
  • Liquid- and echo-state machines use randomly connected recurrent networks as filter bases with tunable readout weights for learning temporal patterns.
  • Training only the readout layer simplifies optimization to regression for supervised learning, while echo-state networks can also be trained by reinforcement learning.

2.2 Biological implementation of optimization

The paper surveys biologically plausible approximations to efficient gradient-based optimization, including perturbation methods, recurrent and bidirectional mechanisms, timing-dependent plasticity, and feedback alignment. Evidence suggests such networks can optimize complex temporal functions, but biological realism and temporal credit assignment remain open issues.

  • Local self-organization alone may be insufficient, motivating biologically plausible mechanisms for efficient gradient computation and optimization.
  • Serial perturbation optimizes by changing one weight at a time and comparing the resulting cost, but it is slow and requires global coordination.
  • Weight perturbation changes all weights simultaneously but generally produces high-variance gradient estimates, while node perturbation can reduce variance.
  • Perturbation methods use local activity and a single global cost signal, but are slowed by serial iteration or low signal-to-noise estimates.
  • Backpropagation rapidly computes sensitivities through layered structure with no gradient-estimate variance for a single input-output pattern.
  • Proposed biological approximations include generalized recirculation, contrastive Hebbian learning, symmetric bidirectional networks, timing-based mechanisms, and feedback alignment.
  • Feedback alignment can approach backpropagation performance on several tasks when synaptic normalization and approximate sign-concordance are present.
  • Biologically plausible networks may efficiently optimize complex functions of temporal history in realistic spiking networks, although the evidence remains preliminary.

2.3 Alternative mechanisms for learning

Biological learning may rely on mechanisms beyond standard gradient descent, including dendritic computation, neuromodulation, cortical architecture, replay, and Bayesian or active-learning processes.

  • 2.3.1 Exploiting biological neural mechanisms: Dendritic computation could support credit assignment by combining nonlinear processing, activity-dependent feedback, and separate storage of activation and error information.Biological models have proposed using dendritic compartments to implement backpropagation-related computations.
  • 2.3.1 Exploiting biological neural mechanisms: Neuromodulators can change circuit responses and plasticity according to global state, expanding the range of optimization algorithms available to the brain.The text highlights dopamine, serotonin, norepinephrine, acetylcholine, and neuropeptides as diverse modulatory substrates.
  • 2.3.2 Structured cortical learning: Cortical laminar structure and local inhibitory networks may shape learning through prediction, feedback control, competition, and gated state transitions.These models connect cortical architecture to unsupervised prediction and Bayesian message passing while assigning control roles to local circuits.
  • 2.3.3 One-shot learning: One-shot learning may use invariant templates, replay, Bayesian sequential-program learning, or interactions among multiple brain areas rather than simple gradient descent.Specialized attention, feedback, and memory mechanisms can also support some backpropagation-based one-shot generalization.
  • 2.3.4 Active learning: Active learning could select informative inputs by representing uncertainty and choosing samples that reduce uncertainty or improve prediction and compression.The passages also note that adaptive sampling can itself be learned through gradient descent or reinforcement learning.

2.4 Differing biological requirements for supervised and reinforcement learning

Supervised and reinforcement learning differ in their teaching signals, and reinforcement learning requires additional mechanisms for deep credit assignment. Deep reinforcement-learning methods provide a partial bridge to biologically plausible learning, but long timescales remain difficult.

  • 2.4 Differing biological requirements for supervised and reinforcement learning: Supervised learning supplies an output error vector, whereas reinforcement learning provides a more limited evaluation signal rather than a full target pattern.This difference makes deep reinforcement learning biologically more demanding than supervised learning.
  • 2.4 Differing biological requirements for supervised and reinforcement learning: Deep Q-learning trains most of a deep network through backpropagation while estimating future reward through a learned Q function.Experience replay and specialized mini-batch procedures contribute to efficient and stable learning.
  • 2.4 Differing biological requirements for supervised and reinforcement learning: Temporally local information makes this target-generation process relatively compatible with known nervous-system constraints.The method therefore bridges reinforcement learning and efficient backpropagation-based gradient descent.
  • 2.4 Differing biological requirements for supervised and reinforcement learning: Long timescales remain a key reinforcement-learning problem because learning coherent behavior over extended sequences weakens formal optimality guarantees.The text contrasts this limitation with human use of prospective memory and discusses hierarchical procedures as a possible direction.
  • 2.4 Differing biological requirements for supervised and reinforcement learning: The brain may use deep reinforcement learning not only for global rewards but also to train local networks with diverse internally generated cost functions.Specialized memories, replay, and hierarchical control could reduce reliance on powerful credit-assignment mechanisms.

3. The cost functions are diverse across brain areas and time

The paper proposes that the brain uses diverse, internally generated cost functions rather than a single end-to-end objective. These functions can differ across regions, learning types, and developmental time, guiding data-efficient acquisition of biologically relevant features.

  • Internally generated cost functions may support supervised, unsupervised, reinforcement-based, or hybrid learning for particular functions.
  • Temporal coherence, prediction, and reconstruction provide alternative unsupervised objectives for learning useful representations.
  • Reinforcement-learning mechanisms may train local networks with internally generated error signals, beyond standard unsupervised learning.
  • Bootstrap cost functions provide hints or heuristic biases that reduce data requirements while avoiding fully unsupervised learning.

4. Optimization occurs in the context of specialized structures

The paper argues that optimization alone is insufficient for complex cognition: the brain also requires specialized, pre-structured architectures matched to different computational problems. These architectures must coordinate data and training signals across regions and times.

  • Blank-slate optimization is argued to be insufficient for complex cognition, even with diverse genetic cost functions and local learning rules.
  • Specialized systems may support planning, sequential multi-step processing, memory access, and compositional or recursive structure manipulation.
  • Training optimization modules may require dynamic coordination of learning rules and target, data, reinforcement, attention, and modulatory signals.
  • The brain combines conserved structures such as cortex with specialized structures including thalamus, hippocampus, basal ganglia, and cerebellum.
  • Cortical microcircuits may vary across areas, although the degree of architectural specificity remains under debate.
  • The proposed mapping from computational problems to brain structures is preliminary and the origins of particular processes remain unsettled.

4.1 Structured forms of memory

The paper treats memory as a collection of structured systems rather than a single mechanism. It distinguishes long-term associative retrieval, working memory, implicit learned representations, and saliency-linked storage and access.

  • 4.1 Structured forms of memory: Long-term memory must support retrieval across contexts, including content-addressable access to memories stored in related situations.
  • 4.1 Structured forms of memory: Working memory is short-term, limited, and rapidly rewritable, whereas implicit memory is sculpted into neural network weights.
  • 4.1.1 Content addressable memories: Content-addressable memories complete stored patterns from partial or noisy cues and can support database-like retrieval.
  • 4.1.1 Content addressable memories: Hippocampal CA3 is proposed as an auto-associative memory, with pattern separation occurring in the dentate gyrus.
  • 4.1.2 Working memory buffers: Whether the brain uses differentiable addressable memories or gating mechanisms based on weighted memory superpositions remains unclear.
  • 4.1.2 Working memory buffers: Some working-memory models attribute maintenance to persistent, self-reinforcing activity in recurrent prefrontal networks.
  • 4.1.2 Working memory buffers: Prefrontal working memory appears to contain multiple functionally distinct subsystems.
  • 4.1.3 Storing state in association with saliency: Information can be stored and retrieved selectively using intrinsic properties or appended tags, creating a route to more complex tasks.

4.2 Structured routing systems

The paper proposes structured routing systems that selectively direct information and training signals across brain regions. Attention improves learning efficiency, while gating mechanisms may control long-range transfer between specialized buffers and areas.

  • Structured routing systems must operate across multiple temporal and spatial scales and support multiple modalities of control.
  • 4.2.1 Attention: Attention can allocate computation to selected objects, reduce learning sample complexity, and support programmatic control of multi-step perceptual tasks.
  • 4.2.1 Attention: Saliency can tag important memories, remove uninformative training data, and guide replay or generative sampling toward useful examples.
  • 4.2.1 Attention: The mechanisms controlling attention allocation and its physical mediation remain active questions, complicated by multiple attention types.
  • 4.2.2 Memory buffers: Memory buffers can store variables abstractly, allowing operations to generalize across different contents that fill the same buffer.
  • 4.2.3 Discrete gating of information flow between buffers: Dynamic routing is needed because dense connectivity is local, while arbitrary distant communication risks crosstalk.
  • 4.2.3 Discrete gating of information flow between buffers: Basal-ganglia and cortical systems may gate information transfer between cortical areas through thalamic relays and action-selection mechanisms.
  • 4.2.3 Discrete gating of information flow between buffers: Dopamine-related activity is proposed to support temporal-difference reinforcement learning of gating policies.

4.3 Structured state representations to enable efficient algorithms

The brain may use specialized and hierarchical structures to represent states and solve computational problems efficiently. These structures support control, navigation, planning, recursion, and other tasks by matching representations to problem structure.

  • 4.3.1 Structured representations: The brain may be pre-structured to create representations suited to particular algorithmic problems rather than relying solely on gradient descent.Grid-like representations are one example of structures that can facilitate spatial computation.
  • 4.3.1 Structured representations: The cerebellum supports continuously changing models of the body and environment, potentially using error signals to update these models in real time.Its possible role extends beyond motor control to cognitive problems that share computational features with movement.
  • 4.3.2 Hierarchical control: Control problems may be organized hierarchically, with specialized levels handling muscles, limb movements, local tasks, and long-term planning.Different levels may optimize distinct problems using their available training signals.
  • 4.3.2 Hierarchical control: Higher-level control may use abstract syntax, chunked action sequences, and goal hierarchies to plan nested sequences of behavior.Human reward systems reflect changing goals and subgoals, while hierarchical reinforcement learning incorporates related representations.
  • 4.3.2 Hierarchical control: The cost functions that create useful hierarchical decompositions of control problems remain unknown.This open question concerns how complex control tasks are divided into component problems.
  • 4.3.3 Spatial planning: The hippocampus may support efficient spatial learning and navigation, while hippocampal-prefrontal interactions could extend navigation to abstract spaces of goals and subgoals.Specialized path-finding structures may simplify higher-level prospective planning.
  • 4.3.4 Recursion: Dynamic hierarchies are needed for arbitrary nested references, including the recursive composition of linguistic forms.Fixed cortical hierarchies alone may not implement recursion on the fly.

4.4 Other specialized structures

Neuroscience contains many specialized structures that may contribute to higher cognition and learning. Their distinct learning rates and coordinated operations provide examples of systems beyond generic data-driven optimization.

  • 4.4 Other specialized structures: Specialized structures including the thalamus, cerebellum, amygdala, retina, hippocampus, basal ganglia, and cortex may support distinct computational roles.The paper highlights routing, rapidly trainable control, utility functions, and other specialized functions.
  • 4.4 Other specialized structures: The paper emphasizes that specialized brain systems receive less attention than their possible importance for higher cognition warrants.This is presented as a scope of emphasis rather than a settled functional conclusion.
  • 4.4 Other specialized structures: The hippocampus may enable rapid learning while the cortex learns more slowly, supporting one-shot acquisition of new concepts through coordinated sequential computation.The proposed process involves the basal ganglia, hippocampus, and prefrontal cortex.

4.5 Relationships with other cognitive frameworks involving specialized systems

The cost-function framework is presented as complementary to cognitive theories that emphasize compositionality, causality, and learning to learn. Structured, dynamically recombinable neural modules are proposed as one route for implementing these capabilities.

  • 4.5 Relationships with other cognitive frameworks involving specialized systems: The framework complements proposals for generative world models with compositional representations, causal inductive biases, intuitive physics, and learning to learn.These ideas motivate neural systems that go beyond purely data-driven optimization.
  • 4.5 Relationships with other cognitive frameworks involving specialized systems: Modular networks whose units and representations can be dynamically combined, copied, and routed may improve compositionality and transfer learning.Their control flow could be learned through reinforcement learning.
  • 4.5 Relationships with other cognitive frameworks involving specialized systems: It remains open whether these cognitive mechanisms should be implemented through cost functions and training procedures, specialized computations, or both.The paper frames implementation across these levels as a key computational problem.
  • 4.5 Relationships with other cognitive frameworks involving specialized systems: More complex primitives, such as state machines with message passing or networks implementing Bayesian inference, could provide useful structured computations.The paper suggests that some such primitives may be found in the brain.

5. Machine learning inspired neuroscience

The paper proposes experimental strategies for testing whether and how brains optimize cost functions, while stressing that optimization may occur at multiple scales and coexist with other learning mechanisms. Detailed brain maps remain essential for identifying training signals, optimization units, and specialized circuitry.

  • 5. Machine learning inspired neuroscience: Neuroscience can test optimization hypotheses by comparing measured neural representations with simulations optimized under candidate cost functions.This endpoint strategy requires knowledge of environmental statistics and candidate objectives.
  • 5. Machine learning inspired neuroscience: Learning dynamics can be assessed by testing whether synaptic-weight changes contain a strong gradient-descent component and weak rotational component.This would require monitoring and externally manipulating synaptic strengths over time.
  • 5. Machine learning inspired neuroscience: Perturbation experiments can test whether learning restores disrupted optimality and whether synaptic reorganization follows predictions from candidate cost functions.Small perturbations might return the system to the same local minimum.
  • 5. Machine learning inspired neuroscience: Brain-machine interfaces could impose user-defined local learning problems if circuits delivering error signals and input data were identified.Connectomic maps may reveal infrastructure supporting both computation and cost-function optimization.
  • 5. Machine learning inspired neuroscience: The critical unresolved question is the unit and scale of optimization, from individual neurons and microcircuits to larger modules or the whole brain.Different scales could support simultaneous local and end-to-end optimization.
  • 5. Machine learning inspired neuroscience: Detailed structure-function inference is difficult because identical initial networks can learn different functions under different training procedures.This complicates reverse engineering based only on connectivity and dynamics.
  • 5. Machine learning inspired neuroscience: Brain maps should be used to determine how optimization is implemented, where training signals originate, and which specialized structures participate.The paper argues that optimization does not make detailed measurements of connectivity and dynamics unnecessary.

6. Neuroscience inspired machine learning

Neuroscience may transform machine learning by revealing how brains achieve general intelligence with little data through diverse optimization processes and specialized structures. The paper highlights brain-inspired directions for efficient, broad learning.

  • The brain’s ability to exhibit intelligent behavior across broad problems using relatively little data makes neuroscience a potential source of machine-learning advances.
  • Brain-inspired optimization could improve machine-learning hardware, learning rules, and architectures across energy efficiency, nonlinear spiking systems, and few-shot active learning.
  • Theories of how the brain performs optimization remain preliminary, making multi-layer credit assignment and possible biological backpropagation important empirical questions.
  • Diverse training problems and cost functions could shape visual representations in series and parallel, enabling networks to be shared, augmented, retrained, or used as critics.
  • Determining the developmental order of biological cost functions could inform divide-and-conquer learning systems, active learning, and hierarchical behaviors.
  • Specialized and modular architectures, including memory and reinforcement-learning systems, are becoming mainstream and may offer clues for combining the brain’s diverse structures.

7. Did evolution separate cost functions from optimization algorithms?

The paper hypothesizes that brains separate general optimization mechanisms from area-specific cost functions. This separation could let evolution improve shared optimization while tailoring computation to different cortical problems.

  • Deep learning separates distributed optimization from designing task-specific cost functions, allowing new applications without changing the optimization algorithm.
  • The authors hypothesize that brain circuits similarly separate optimization mechanisms from cost functions that differ across cortical areas.
  • Separating optimization and cost-function generation could let evolution produce better computations faster by combining common unsupervised mechanisms with area-specific learning signals.
  • One proposed implementation keeps some wiring and learning rules relatively fixed while making other elements that specify cost functions more variable across areas.

8. Conclusions

The paper argues for theoretical frameworks that guide neuroscience from broad principles toward neural details, while emphasizing heterogeneous optimization and specialized structures. It presents dialogue with machine learning as a route toward more efficient general intelligence.

  • Because bottom-up neural-data analysis faces interpretation challenges, theoretical frameworks can constrain hypotheses before researchers examine detailed mechanisms.
  • A dialogue between neuroscience and machine learning could help clarify whether differing cost functions provide a useful framework for understanding neural computation.
  • Brain optimization may inform machine learning through energy-efficient hardware, heterogeneous developmental cost functions, and specialized structures for learning with little supervision.
  • The proposal rejects single mechanisms, cost functions, representations, and homogeneous structures in favor of many interacting specialized elements.
  • The proposal is presented as an argument against simple end-to-end training and for heterogeneous systems containing computationally specialized structures.
  • The framework portrays intelligence as many specialized structures trained with developmentally regulated cost functions that evolution also optimizes.
Loading 1606.03813v1…