PaperlayerView exact revision on arXiv

Source-linked AI summary

Linking connectivity, dynamics and computations in low-rank recurrent neural networks

Francesca Mastrogiuseppe, Srdjan Ostojic

arXiv:1711.09672v2q-bio.NC

TL;DR

The paper asks how low-dimensional computations over mixed neural representations emerge from recurrent connectivity and inputs. It studies networks combining random and low-rank structure, using connectivity geometry to infer dynamics and design computations. The dynamical repertoire and computational capacity increase quickly with connectivity dimensionality, while the analysis is derived under large-network and weak-structured-connectivity assumptions.

  • Problem

    Determining how low-dimensional computations and dynamics emerge from recurrent connectivity and inputs remains a major challenge in relating connectivity, activity, and behavior.

  • Method

    The authors study recurrent networks whose connectivity sums a random matrix and a controlled low-rank structure, analyzing dynamics through the geometry of connectivity vectors and inputs.

  • Results

    Dynamical repertoire increases quickly with connectivity dimensionality, and rank-two connectivity structures suffice for complex context-dependent tasks.

  • Takeaways & Limitations

    Connectivity structure can provide a minimal scaffold for low-dimensional dynamics and specific computations while random connectivity contributes heterogeneous fluctuations.

  • Takeaways & Limitations

    The effective statistical analysis is derived for large networks with weak low-dimensional connectivity whose entries scale inversely with network size.

Abstract

from arXiv · show

Large scale neural recordings have established that the transformation of sensory stimuli into motor outputs relies on low-dimensional dynamics at the population level, while individual neurons exhibit complex selectivity. Understanding how low-dimensional computations on mixed, distributed representations emerge from the structure of the recurrent connectivity and inputs to cortical networks is a major challenge. Here, we study a class of recurrent network models in which the connectivity is a sum of a random part and a minimal, low-dimensional structure. We show that, in such networks, the dynamics are low dimensional and can be directly inferred from connectivity using a geometrical approach. We exploit this understanding to determine minimal connectivity required to implement specific computations, and find that the dynamical range and computational capacity quickly increase with the dimensionality of the connectivity structure. This framework produces testable experimental predictions for the relationship between connectivity, low-dimensional dynamics and computational features of recorded neurons.

Introduction

The paper addresses the difficult relationship between recurrent connectivity, neural activity, and behavior by studying networks that combine random connectivity with minimal low-rank structure. This framework links connectivity geometry to low-dimensional dynamics and increasingly complex computations.

  • Motivation: Understanding causal relationships among synaptic connectivity, neural activity, and behavior remains challenging even with complete measurements.The paper identifies the lack of an appropriate conceptual framework as a central obstacle.
  • Motivation: Fully random recurrent networks generate irregular activity but produce stereotyped responses, limited input-output computations, and typically high-dimensional dynamics.These properties motivate models with additional structured connectivity.
  • Motivation: Several functional approaches converge on minimal low-dimensional connectivity structures whose matrices are low rank rather than fully specifying every connection.This pattern appears across multiple computational network designs.
  • Approach: The studied networks combine structured low-rank and random connectivity, enabling dynamics predicted from geometric relationships among connectivity vectors and feed-forward inputs.The framework uses a small number of high-dimensional vectors to characterize the relevant geometry.
  • Key result: Dynamical repertoire increases quickly with connectivity dimensionality, with rank-two structures sufficient for complex context-dependent tasks.The paper examines four tasks ranging from binary discrimination to context-dependent evidence integration.

Results

Low-rank connectivity organizes spontaneous and stimulus-evoked activity into low-dimensional dynamics that are geometrically related to connectivity and input vectors. Increasing structural rank expands the dynamical repertoire, enabling computations from simple discrimination to context-dependent tasks.

  • Network model: The model combines a fixed, weak, structured low-rank matrix with an uncontrolled random connectivity matrix.The structured matrix has entries of order 1/N and is specified by a small number of independent rows and columns.
  • Spontaneous dynamics: Nonzero connectivity overlap organizes spontaneous activity along the right-connectivity vector m, while stronger random connectivity adds orthogonal heterogeneity and can induce chaos.Weak structure yields unstructured activity, whereas stronger structure produces heterogeneous activity and, under the stated symmetry, bistability.
  • Stimulus-evoked dynamics: When m and n are orthogonal, input overlap with n can reveal the connectivity structure and progressively amplify activity along m.When m and n instead share a sufficiently strong overlap, inputs along n can suppress one bistable branch and produce a nonlinear transformation.
  • Stimulus-evoked dynamics: External inputs suppress chaotic and bistable dynamics, reducing the prevalence of these spontaneous regimes.This suppression is independent of the specific geometrical configuration between the input and connectivity vectors.
  • Stimulus-evoked dynamics: External inputs generally produce two-dimensional population trajectories in the plane spanned by the right-connectivity vector m and input vector I.The vector m determines the output pattern, while the left-connectivity vector n selects inputs amplified along m; principal-components analysis confirms these dominant dimensions.
  • Computations: Rank-r connectivity confines average activity to an (r + 1)-dimensional subspace and allows rank-two structure to implement context-dependent computations.The framework links connectivity, low-dimensional dynamics, and computation; in the Go-Nogo task, choosing m = w and n = IA implements selective readout and predicts stimulus-dependent dimensionality.

Discussion

Low-rank recurrent connectivity links connectivity geometry to low-dimensional dynamics and enables minimal implementations of input-output computations. The framework also yields experimentally testable predictions, while remaining limited by simplified network assumptions.

  • Core framework: Low-rank structure induces low-dimensional dynamics that can be predicted directly from connectivity and input geometry.The approach uses mean-field theory and supports geometrical interpretations of network dynamics.
  • Computational capacity: Rank-two connectivity structures already support a variety of complex, context-dependent computations.The study examines four tasks of increasing complexity, ending with context-dependent evidence integration.
  • Distributed representations: Random low-rank structures represent stimuli and outputs along heterogeneous, highly distributed directions rather than assigning neurons to single task features.This contrasts with clustered connectivity, where neurons are highly specialized and exhibit pure selectivity.
  • Computational roles: Left- and right-connectivity vectors have distinct computational roles: the left vector selects inputs, whereas the right vector determines network output.The paper focuses on responses to external inputs and input-output computations rather than item memorization.
  • Limitations: The model uses simplified rate-unit dynamics without positive firing rates, excitation-inhibition segregation, or spike-based interactions.Implementing the framework in spiking networks requires additional work.
  • Experimental predictions: The framework predicts relationships among connectivity, low-dimensional dynamics, and individual-neuron computational properties that can be tested experimentally.For recurrently generated dynamics, neurons loading strongly on dominant components are predicted to have stronger-than-average mutual connections.

The network model

The model uses large recurrent rate networks whose connectivity combines a fixed low-rank structure with an independent random component. Higher-rank structures are represented by multiple pairs of connectivity vectors.

  • Units and dynamics: Each unit is a rate unit with continuous activation and a nonlinear output, modeled primarily with φ(x) = tanh(x).The transformed activation is interpreted as the unit’s firing rate.
  • Connectivity structure: Connectivity is the sum of a Gaussian all-to-all random matrix and a fixed low-rank matrix.The random component varies across realizations, while the structured component remains fixed.
  • Rank-one structure: For rank one, the structured connectivity is the external product of two vectors, m and n.These are the right- and left-connectivity vectors, respectively.
  • Rank-one structure: The rank-one structure has one nonzero eigenvalue, mT n/N, which defines the connectivity strength.The corresponding right and left eigenvectors are m and n.
  • Higher-rank structures: A rank-r structure is specified by r linearly independent pairs of right- and left-connectivity vectors.The framework assumes r is much smaller than the network size N.

Overview of Dynamical Mean-Field theory

Dynamical Mean-Field theory reduces the large network to self-consistent macroscopic equations whose dimensionality follows the rank of the structured connectivity. The resulting activity occupies a low-dimensional space shaped by connectivity vectors and inputs.

  • DMF reduction: DMF theory averages over random connectivity to derive an effective stochastic description for each unit.The random contribution is approximated by a Gaussian stochastic process.
  • Single-unit description: Each unit follows a Langevin-like equation, with effective inputs characterized through their first- and second-order statistics.Different units are coupled only through these population-level statistics.
  • Rank-one dynamics: For rank-one connectivity, the mean input depends on the overlap between the left-connectivity vector and average population activity.The overlap is denoted κ and provides a geometrical summary of the network state.
  • Macroscopic closure: Stationary rank-one dynamics are determined by two coupled macroscopic equations, while temporally fluctuating dynamics require three nonlinear equations.These equations describe population-level quantities such as overlap and variance.
  • Readout: The network’s linear readout is the projection of average firing rates onto the readout vector w.Its expression parallels the connectivity overlap after replacing n with w.
  • Higher-rank dynamics: For rank r, activity lies in an (r + 1)-dimensional space spanned by the r right-connectivity vectors and the input vector.The corresponding DMF description uses r overlaps and r + 1 coupled nonlinear equations.

Details of Dynamical Mean-Field theory

The detailed DMF analysis derives self-consistent effective-noise statistics, stationary states, and stability criteria. Stability is governed by a random spectral component and structured outliers, which produce distinct dynamical regimes.

  • Effective noise: DMF assumes that random connectivity decorrelates single-neuron activity in sufficiently large networks, and the analysis verifies this self-consistently.The effective inputs are found to be uncorrelated across different units.
  • Effective noise: The effective-input mean depends on the population overlap κ and varies across units through their right-connectivity components.Its autocorrelation is identical across units.
  • Stationary dynamics: Stationary solutions are obtained by solving the Langevin-like process, with mean and variance determined self-consistently from population averages.The structured connectivity shapes average activity along its right-eigenvector direction.
  • Stability spectrum: The connectivity eigenspectrum combines a continuous random component inside a circle with structured outlier eigenvalues.The outlier is induced by the low-rank component.
  • Homogeneous stationary solutions: A homogeneous stationary solution loses stability when either mT n/N or g exceeds 1/φ′(x̄).The random instability generates irregular temporal activity, whereas an outlier instability develops along one structured direction.
  • Transient dynamics: The largest decay timescale predicts an outlier’s position and retains a structured eigenvector component aligned with m.This component persists despite averaging over random connectivity realizations.
  • Stability analysis: Mean-field predictions of the spectral radius and outlier assess stability for arbitrary stationary solutions.The two instability components are expected to produce qualitatively different dynamical regimes.

Dynamical Mean Field equations for chaotic solutions

The DMF framework characterizes temporally fluctuating activity through mean activity, overlap, and two variance measures, while relating stability boundaries to the structured connectivity and random spectrum.

  • Autocorrelation: The full autocorrelation function ∆(τ) is analyzed through an effective-noise description and a one-dimensional potential whose asymptote is ∆∞.Monotonic solutions satisfy a stationary-point condition at ∆∞ and an energy-conservation condition.
  • State variables: Temporal fluctuations are quantified by the variance difference ∆0 − ∆∞, which vanishes when activity is stationary.The fluctuating-state equations reduce to the stationary DMF description when ∆0 = ∆∞.
  • Structured dynamics: The structured connectivity shapes activity along its right eigenvector when the overlap κ is nonzero, and this can permit multiple chaotic solutions.Numerical simulations suggest chaotic states inherit stability properties from the stationary states from which they develop, although a formal stability analysis was not performed.
  • Chaotic onset: At the critical point for chaotic fluctuations, temporal variance disappears, so ∆0 = ∆∞, and the potential changes concavity.These conditions yield the instability criterion used to locate the stationary-to-chaotic boundary.
  • Stationary solutions: Stationary solutions arise from intersections of the µ and ∆0 nullclines, with their number and type controlled by structure strength MmMn and disorder strength g.For MmMn > 1, symmetric heterogeneous states appear; for g > 1, an additional zero-mean heterogeneous solution can occur.

Chaotic solutions

Chaotic regimes emerge when disorder destabilizes stationary activity, while structured connectivity and external inputs determine whether chaotic states are homogeneous, heterogeneous, bistable, or hybrid.

  • Chaotic phases: For g > 1 and MmMn < 1, one homogeneous chaotic state exists; crossing MmMn = 1 produces two symmetric heterogeneous chaotic states.The central state has zero temporally averaged activity, whereas the symmetric states have nonzero µ and ∆∞.
  • Chaotic phases: The critical disorder boundary gB is obtained by linearizing the DMF dynamics around the central solution and finding when the largest eigenvalue reaches unity.This defines the boundary where heterogeneous chaotic states emerge.
  • Connectivity structure: Nonzero structure strength can generate heterogeneous stationary states, while sufficiently strong disorder favors homogeneous or unstructured dynamics.For orthogonal structures without inputs, all outlier eigenvalues vanish and a single homogeneous attractor is stable.
  • External inputs: External inputs orient activity through the overlap κ, break sign-reversal symmetry, and can destabilize or eliminate one member of a bistable pair.An input correlated with n can increase κ or cause one stable solution to annihilate with an unstable one.
  • External inputs: Modulatory inputs along the structure-overlap direction control detection thresholds and can suppress responses to irrelevant stimuli for context-dependent readouts.The same mechanism is used to regulate nonlinear task responses and context-dependent outputs.
  • External inputs: With external inputs, distinct stability boundaries can produce hybrid regimes in which a static solution coexists with a chaotic attractor.Input-induced asymmetry requires positive and negative fixed points to be assessed separately.
  • Null overlap: For orthogonal connectivity vectors, the network is silent without inputs and has no outlier eigenvalues because every eigenvalue of Pij vanishes.The unique stable attractor may be stationary or chaotic depending on random connectivity strength.

Rank-two structures with internal pairwise overlap

Rank-two structures with internal overlap support degenerate rings of activity, slow marginal dynamics, and oscillatory or bistable regimes whose symmetry is controlled by overlap geometry and inputs.

  • Connectivity geometry: The two nonzero connectivity eigenvalues correspond to the right vectors m(1) and m(2), so instability tends to occur along the direction with the strongest overlap.This links the eigenspectrum directly to the geometry of the rank-two structure.
  • Degenerate rank-two states: When two rank-two overlap strengths are equal, the mean-field equations constrain only κ1^2 + κ2^2, producing a one-dimensional ring of solutions.The ring radius depends on the relative magnitudes of structured overlap and disorder, and strong disorder suppresses nontrivial states.
  • Degenerate rank-two states: The ring is a continuous set of marginally stable states, with one outlier eigenvalue pinned to the stability boundary and associated slow time scales.Finite networks settle near stable points on the ring, and activity explores the resulting slow manifold.
  • Input perturbations: Inputs orthogonal to both left vectors preserve the ring degeneracy but reduce its radius, whereas inputs overlapping either left vector collapse the ring to a unique stable point.The overlap component breaks the degeneracy by uniquely specifying κ1 and κ2.
  • Oscillatory regimes: For γ > 2, real eigenvalues produce bistable stationary activity, while for γ < 2, complex eigenvalues can generate nonlinear slow oscillations near the annihilation boundary.At γ = 2, the stable and unstable stationary pairs annihilate and disappear.
  • Oscillatory regimes: A flat phase distribution can arise when oscillator real and imaginary parts are independently distributed, and can also be produced by a rank-two structure with internal overlap.Thus, phase organization depends on the geometry of the structured connectivity.
  • Oscillatory regimes: As γ approaches 2, oscillatory activity in κ1 and κ2 becomes strongly anti-phase, with phases tending toward opposite values.The phase distribution sharpens as γ approaches 2.

Implementation of computational tasks

The networks implement discrimination, threshold modulation, and context-dependent computations through low-rank connectivity, nonlinear dynamics, and stimulus or modulatory inputs.

  • Go-Nogo discrimination: The Go-Nogo task uses orthogonal Gaussian stimulus patterns, linear readout, and connectivity vectors aligned so the network selectively responds to Go.The right-connectivity vector matches the readout, while the left-connectivity vector matches the preferred Go stimulus.
  • Go-Nogo discrimination: The initialized negative κ state represents Nogo, whereas sufficiently strong preferred input drives the network to the positive κ Go state.An offset sets the baseline Nogo output to zero.
  • Continuous stimulus detection: The detection threshold decreases toward zero as structure strength ρmρn decreases from 4, while arbitrarily large time-scales occur near the bistability critical point.The threshold and dynamical time-scale are therefore controlled by rank-one structure strength.
  • Contextual modulation: A modulatory input overlapping the left-connectivity vector acts as a constant offset to stimulus strength, shifting the response curve and regulating the detection threshold.The modulation strength γ determines the shift.
  • Context-dependent computations: Rank-two connectivity generates a continuous ring attractor, while a common overlap direction destabilizes it into two symmetric stable fixed points.The resulting positive and negative κ1, κ2 states support Nogo and Go conditions.
  • Context-dependent computations: Context directions gate responses: strong negative gating can suppress the IA response, leaving the readout responsive to IB, whose threshold is close to 0.5.Different IA amplitudes minimally change this threshold, while excessive input noise disrupts fully context-dependent responses.

Method details for Main Figures

The main figures use finite recurrent networks with fixed low-rank structure, varied random connectivity, and Gaussian inputs across multiple realizations.

  • Simulation design: Simulations vary random connectivity across network realizations while keeping the low-rank component fixed.Reported ensembles include 20 networks of 3500 or 2500 units and 50 networks of 7500 units.
  • Input and readout conditions: Input traces are generated with specified means, standard deviations, and context amplitudes, and trajectories are sometimes Gaussian-smoothed for display.The threshold is indicated in the response plots, while readout offsets set baseline values to zero.

Quantification and Statistical Analysis

The analysis combines PCA of population activation with multivariate regression to quantify low-dimensional dynamics and single-unit tuning to task variables.

  • Dimensionality reduction: PCA extracts the low-dimensional subspace from activation matrices whose columns are trial-averaged unit time traces.Activation matrices are constructed across network simulations with noisy inputs or quenched connectivity noise.
  • Dimensionality reduction: The activation matrix is transformed into principal-component coordinates, and each component’s explained variance is obtained from the rotated correlation matrix.The PCs are eigenvectors of C = X^T X, ordered by decreasing eigenvalue.
  • Robustness: The qualitative conclusions remain valid when PCA is applied to nonlinear activation variables φ(xi), which form a dominantly low-dimensional manifold.The quantitative results depend on whether activation variables are Z-scored or mean-subtracted.
  • Dimensionality reduction: The low-rank connectivity produces purely low-dimensional dynamics, whereas random connectivity contributes a continuum of components whose amplitude depends on g.Averaging across 20 random-connectivity realizations reveals the low-dimensional structure when g = 0.8.
  • Regression analysis: Regression estimates how each neuron’s activation depends on task variables, using least-squares inversion of a task-feature matrix across trials.Coefficients are selected at the time point where each unit’s coefficient-vector norm is maximal and assembled into population regression axes.
  • Task-variable definitions: The analyzed task variables include Go and Nogo strengths, stimulus strength and output, and stimulus strengths, context, and output.Context is encoded as y = 1 for Context A and y = −1 for Context B.

D E F

The supplementary analyses examine mean-field dynamics, stability spectra, stationary solutions, and two-dimensional input-driven activity for rank-one networks.

  • D: Figure S1 analyzes rank-one networks whose right- and left-connectivity vectors overlap only along the unitary direction using Dynamical Mean-Field theory.It is related to Figure 1 and uses the ρ = 0 condition.
  • E: The stability analysis compares finite-size eigenspectra with mean-field predictions for the random bulk and structured outlier eigenvalue.The outlier position is evaluated through a mean-field stability analysis rather than the naive structured-matrix eigenvalue.
  • F: Graphical analyses use nullclines of population-averaged DMF equations, marking solutions stable or unstable with respect to the outlier eigenvalue.Nullcline shapes depend on structure strength MmMn and disorder strength g.
  • D: Figure S2 extends the rank-one analysis to connectivity vectors overlapping along an arbitrary direction y with Mm = Mn = 0 and ρ ≠ 0.This supplementary analysis is also related to Figure 1.
  • F: Figure S3 presents two-dimensional dynamics in unit-rank networks with external inputs, while Figure S4 gives a related input-driven DMF description.The supplementary analyses connect external-input responses to the low-rank structure.
  • F: The external-input DMF analysis includes overlap between the input vector and n on the unitary direction plus orthogonal input components.One example uses MI = 0.13 and ΣI = 0.3.

M L N

The supplementary results characterize low-rank network dynamics across finite-size, chaotic, and computational settings, including bistability, oscillations, and context-dependent readouts.

  • Ring attractor: Rank-two connectivity structures with strong internal overlaps produce a ring attractor in stationary finite-size networks and can coexist with chaotic fluctuations.The stationary example uses g = 0.5, whereas the chaotic example uses g = 2.1.
  • Context-dependent detection: Rank-two structures implement context-dependent nonlinear stimulus detection through two activation statistics whose sum is approximately constant on each gating branch.The readout sums κ1 and κ2, while normalized prediction error is evaluated across network sizes.
  • Bistable dynamics: Strong low-rank structure yields bistable states whose chaotic activity has stationary population-averaged statistics and sharp transitions between positive and negative first-order values.Finite-size fluctuations support transitions between the bistable states.
  • Additional regimes: Unit-rank networks with positively defined activation functions and rank-two structures with cross overlaps are examined as distinct dynamical regimes related to the network's low-rank organization.The supplementary figures associate the unit-rank case with Figure 1 and the cross-overlap case with oscillatory activity.
Loading 1711.09672v2…