PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural optimal feedback control with local learning rules

Johannes Friedrich, Siavash Golkar, Shiva Farashahi, Alexander Genkin, Anirvan M. Sengupta, Dmitri B. Chklovskii

arXiv:2111.06920v1q-bio.NCcs.NEcs.SYeess.SY

TL;DR

Neural models of optimal feedback control have not adequately combined delayed sensory feedback, state estimation, and control under online learning constraints. Bio-OFC pairs adaptive Kalman filtering with model-free control in a biologically plausible network, and the authors report that it can build state representations and deal with sensory delays. The paper identifies its linear-control framework as a limitation.

  • Problem

    Prior neural proposals often omitted delayed feedback, separated Kalman filtering from control, or required noise covariances or distinct learning phases, despite the challenges noisy and delayed stimuli pose for accurate movement.

    Source: §1 · pp. 1–2

  • Method

    Bio-OFC combines adaptive Kalman state discovery with a model-free control agent in a biologically plausible neural network.

    Source: §1 · p. 2

  • Results

    The authors report that the circuit can build an adequate state representation, deal with sensory delays, and explore actions to execute control strategies.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The network performs closed-loop control while identifying the system and optimizing control, without separate phases for those processes.

    Source: §4.1 · p. 8

    The work is framed within linear control theory, with locally linearized dynamics or additional neural layers suggested as possible routes to generalization.

    Source: §5 · p. 10

Abstract

from arXiv · show

A major problem in motor control is understanding how the brain plans and executes proper movements in the face of delayed and noisy stimuli. A prominent framework for addressing such control problems is Optimal Feedback Control (OFC). OFC generates control actions that optimize behaviorally relevant criteria by integrating noisy sensory stimuli and the predictions of an internal model using the Kalman filter or its extensions. However, a satisfactory neural model of Kalman filtering and control is lacking because existing proposals have the following limitations: not considering the delay of sensory feedback, training in alternating phases, and requiring knowledge of the noise covariance matrices, as well as that of systems dynamics. Moreover, the majority of these studies considered Kalman filtering in isolation, and not jointly with control. To address these shortcomings, we introduce a novel online algorithm which combines adaptive Kalman filtering with a model free control approach (i.e., policy gradient algorithm). We implement this algorithm in a biologically plausible neural network with local synaptic plasticity rules. This network performs system identification and Kalman filtering, without the need for multiple phases with distinct update rules or the knowledge of the noise covariances. It can perform state estimation with delayed sensory feedback, with the help of an internal model. It learns the control policy without requiring any knowledge of the dynamics, thus avoiding the need for weight transport. In this way, our implementation of OFC solves the credit assignment problem needed to produce the appropriate sensory-motor control in the presence of stimulus delay.

1 Introduction

The introduction motivates a neural model of optimal feedback control that can handle noisy, delayed sensory input while learning and controlling online. Bio-OFC combines adaptive Kalman filtering with model-free policy-gradient control using local synaptic learning rules.

1 Introduction

  • Earlier neural proposals often omitted delayed feedback, separated filtering from control, or required noise covariances and distinct learning phases.The introduction also notes scalability, convergence-loop, and biological-plausibility concerns in prior models.Source: §1 · p. 2
  • A global scalar cost signal updates synaptic connections through policy gradients, and the network is tested in episodic tasks.The described tests include a discrete-time double integrator, a hand-reaching task, and a simplified fly simulation.Source: §1 · p. 2
  • Bio-OFC combines adaptive Kalman state discovery with a model-free control agent in a biologically plausible neural network.The network represents the state, prediction error, and control in different neurons.Source: §1 · p. 2
  • The implementation handles delayed sensory feedback without requiring noise covariances, known dynamics, or separate learning and execution phases.Its model-free control agent enables closed-loop control without weight transport.Source: §1 · p. 2

2 Background

The background formulates control as estimating a hidden state from observations and choosing actions to minimize expected cost, then reviews Kalman estimation and feedback control.

2 Background

  • The background reviews classical Kalman estimation and control, using boldface lowercase and uppercase for vectors and matrices and I for the identity matrix.These conventions apply to the section's equations.Source: §2 · p. 3

2.1 Problem formulation

  • The problem formulation models hidden states as evolving under linear dynamics and control, with observations that are linear functions of state and corrupted by Gaussian noise.Because the dynamics are not known a priori, A, B, and C must be learned online while estimating state and minimizing expected cost.Source: §2.1 · p. 3

2.2 Kalman estimation and control

  • With delayed sensory feedback, the Kalman estimator infers the current state from earlier observations and internal-model predictions; the Kalman gain combines those inputs.For quadratic costs, the output feedback policy simplifies to an LQR, whose control gain is found from a matrix Riccati equation.Source: §2.2 · pp. 3–4

3 Neural network representation for optimal feedback control

Bio-OFC uses an internal model to estimate state from delayed sensory feedback, then learns system parameters and control weights through online local plasticity rules. Its control policy is learned with policy gradients rather than requiring known system dynamics.

3.1 Inference

  • For delays greater than one step, Bio-OFC pairs delayed measurements with equally delayed state estimates, reducing predictive performance only modestly versus a biologically unrealistic scheme.The approach adjusts the synaptic delay associated with C to match sensory delay.Source: §3.1 · p. 4

3.2.1 System identification and Kalman gain

  • Online least-squares updates learn A, B, C, and Kalman gain L from prediction errors without requiring noise covariance knowledge.Replacing C^T with L yields local plasticity rules under mild initialization constraints.Source: §3.2.1 · pp. 4–5

3.2.2 Control

  • Policy gradients learn controller weights K by descending expected trajectory cost, using an eligibility trace to assign each time step's cost.The trace tracks past gradients of the log policy; synaptic tags can track correlations between estimated state and control noise.Source: §3.2.2 · p. 5

4 Experiments

The experiments test Bio-OFC on delayed, noisy control tasks, including system identification and control in a double integrator, reaching under force fields, and simplified flight. Across these settings, the reported results show adaptation to changing noise, near-optimal control costs, human-like reaching trajectories, and less overshoot than policy gradient in flight.

4.1 Discrete-time double integrator

  • With changing observation-noise covariance, the learned Kalman gain adapts and prediction error becomes quickly close to optimal, even when gain convergence is slow.The experiment changes W at 2500, 5000, and 7500 episodes while keeping A, B, and C fixed.Source: §4.1 · p. 6, Figure 2 · p. 7
  • Bio-OFC's control cost remains close to the LQG benchmark across measurement delays, and its internal model helps bridge the performance gap by predicting forward.The comparison includes two observation models and contrasts Bio-OFC with LQG, an ANN, and a model-free controller.Source: §4.1 · pp. 6–7, Figure 3 · p. 7
  • Full system identification converges to optimal prediction error, and assuming three latent dimensions for a two-dimensional system does not affect performance.The learned parameters include A, B, C, and L.Source: §4.1 · pp. 7–8
  • Both staged system identification followed by controller learning and simultaneous system identification and control converge to the optimal cost.The simultaneous closed-loop approach requires no separate system-identification and control-optimization phases.Source: §4.1 · p. 8, Figure 4 · p. 8, Figure 5 · p. 9

4.2 Reaching movements

  • In reaching under an external force field, Bio-OFC captures human trajectory characteristics during training, with adaptation driven primarily by estimator changes.The task uses a 50-ms sensory delay; disabling controller learning yields virtually identical results.Source: §4.2 · p. 9

4.3 Simplified winged flight

  • In simplified flight with 100-ms sensory delay and L1 costs, Bio-OFC flies directly toward the target without significant overshoot, unlike policy gradient, which overshoots and backtracks.The L1 cost tests the algorithm under a cost that differs from the quadratic assumption of LQR.Source: §4.3 · p. 10

5 Discussion

Bio-OFC combines neural state estimation with model-free control to address delayed sensory feedback. Its discussion identifies linear dynamics, uniform delays, and model-free control as important scope boundaries.

5 Discussion

  • The proposed circuit learns a world model with a neurally plausible Kalman filter and uses future-state previews to learn a control policy.The authors say it can represent the appropriate state space, handle sensory delays, and actively explore actions.Source: §5 · p. 10
  • The controller is model-free for simplicity and biological plausibility, while model-based control could offer greater sample efficiency and transfer across goals and tasks.A biologically plausible combination of Kalman state estimation and model-based control is left as a future question.Source: §5 · p. 10
  • The work is limited to linear control theory, with locally linearized dynamics and learned linearizing representations proposed as possible extensions.The discussion presents these as potential generalizations, not results demonstrated in this work.Source: §5 · p. 10
  • The experiments use uniform stimulus delays, although the model can extend to different known modality delays when prediction synaptic delays match them.Learning the appropriate delays using lag vectors is left for future work.Source: §5 · p. 10
  • The authors relate the architecture to proposed brain-region roles: cerebellar system identification, parietal state estimation, cortical control, and basal-ganglia cost or reward processing.These are proposed mappings to regions putatively contributing to optimal feedback control.Source: §5 · p. 10
  • The architecture addresses online control in a changing world with delay and biologically plausible synaptic updates, and performs well on several tasks despite approximations.The authors call for further exploration of its limitations.Source: §5 · p. 10

A Linear-quadratic regulator (LQR)

The section introduces linear-quadratic regulation and its control law, then connects it to Kalman state estimates in linear-quadratic-Gaussian control.

A Linear-quadratic regulator (LQR)

B Experimental details

The experimental-details section reports computing setups and runtimes for figure-generating simulations and hyperparameter optimization; the code section provides reproduction instructions.

B Experimental details

  • The Fig. 2 learning-rate optimization took 98 seconds, and Fig. 3 optimal-gain optimization took seconds to two minutes on a six-core desktop without a GPU.The setup used SciPy minimize with default BFGS on a CentOS desktop with 128 GB RAM.Source: B Experimental details · p. 13
  • Twenty seeded runs of 10,000+1,000 episodes took 9.3 seconds for open-loop and 14.0 seconds for closed-loop control; 1,000 Optuna trials took about 130 and 190 minutes, respectively.Hyperparameter optimization ran on a 14-core cluster node for each LDS, control setting, and delay combination.Source: B Experimental details · p. 13
  • The biological approximations do not change the computational complexity of policy gradient and Kalman filtering.Source: B Experimental details · p. 13

C Code

  • The repository provides figure-reproduction code, requirements, and Optuna hyperparameters, with corresponding scripts saving figures to a subdirectory.Source: C Code · p. 13

D Learning rules

The learning-rule discussion examines how local updates relate to gradient descent and reports their alignment during learning. It also describes conditions and initialization used to support objective improvement.

D Learning rules

  • The marginal mean-squared error decreases when the angle between gradient g_θ and update Δθ is below 90°; multiplying the gradient by a positive-definite matrix preserves positive alignment.The symmetric part of the multiplier determines positive definiteness; its antisymmetric part contributes zero to the quadratic form.Source: D Learning rules · p. 13
  • Replacing C⊤ with L in the learning rules scales gradients by L C⊤+, so C and L are initialized to make that product positive definite.The product need not be symmetric.Source: D Learning rules · p. 13
  • Bio-OFC updates the current state estimate directly from delayed measurements, whereas the alternative forward-predicting ANN would require copies of A and B and memory of past controls.The figure also contrasts a model-based controller using an internal state estimate with a memoryless controller using the currently available delayed observation.Source: Figure S1 · p. 14, Figure S2 · p. 14
  • The reported system-identification plots show 20 runs across two- and three-dimensional observations, including an over-representing model for LDS2.The over-representing Bio-OFC assumes three latent dimensions when the actual system has two.Source: D Learning rules · p. 15
  • The discussion argues that C L is plausibly positive semidefinite when A is near identity and the time discretization is not too coarse.Under A = I, C A Σ C⊤ represents state uncertainty and C Σ C⊤ + W total state-plus-observation uncertainty; C L is treated as their ratio.Source: D Learning rules · p. 15
  • During stochastic gradient descent, individual updates can increase the objective, so gradient alignment is required only on average.Source: D Learning rules · p. 15
  • For LDS2, e_t⊤CL e_t was negative in 7.1% of individual updates, but the episode average was negative in only 0.036% of episodes and positive when averaged across episodes.The simulations do not provide a theoretical derivation that its expectation is positive.Source: D Learning rules · p. 15

E Consequences of replacing C⊤with L

Replacing C⊤ with L yields local learning rules without harming filtering performance. Convergence depends on the initial alignment of C and L: nonpositive λ1 prevents convergence to the optimal asymptotic value.

E Consequences of replacing C⊤with L

  • Replacing C⊤ with L yields local learning rules, with slightly slower convergence and marginally larger average MSE than the Fig. 2 rule, but without its overshoot after a covariance change.The authors conclude that the replacement does not harm performance.Source: E Consequences of replacing C⊤with L · p. 15, Figure S4 · p. 16
  • For λ1 > 0.02, asymptotic average MSE did not differ significantly across tested initial alignments, while convergence was slower near λ1 = 0; for λ1 ≤ 0, performance did not converge to the optimal asymptotic value.The alignment condition is expressed through the eigenvalue of the symmetric part of LC⊤.Source: E Consequences of replacing C⊤with L · pp. 15–16, Figure S5 · p. 16
  • Supplementary sweeps plot open- and closed-loop training cost across controller noise levels and update momenta.The plots cover σ = 0.05, 0.1, 0.2, and 0.5, and m = 0, 0.9, 0.99, and 0.9995.Source: Figure S6 · p. 17, Figure S7 · p. 17, Figure S8 · p. 18

F Learning of a sensory-motor control task

The reaching task models movements in a force field with delayed sensory measurements. Bio-OFC qualitatively matches human reaching behavior and learns trajectories close to straight lines; adaptation remains similar with signal-dependent motor noise and is driven primarily by estimator changes.

F Learning of a sensory-motor control task

  • The reaching model uses a six-dimensional state for position, velocity, and acceleration in x and y, with a 50 ms measurement delay and Cartesian actions.The force-field parameters were chosen for a qualitative match with experimental data.Source: F Learning of a sensory-motor control… · p. 19
  • After about 1000 episodes, learned trajectories are close to straight lines, and updating only the estimator while keeping controller weights fixed yields similar force-field adaptation.The authors attribute this pattern primarily to changes in the estimator, which performs system identification.Source: F Learning of a sensory-motor control… · p. 19, Figure S12 · p. 22
  • With signal-dependent motor noise scaled by the control magnitude, reaching results are similar to those obtained with additive noise.The noise term is multiplied by |u_t| in the plant dynamics.Source: F Learning of a sensory-motor control… · p. 19, Figure S14 · p. 24
  • The supplementary figures present model and human hand trajectories in null and force-field conditions, as well as model prediction error and cost during training.The training trajectories are shown across the first, second, third, and final groups of 250 targets.Source: Figure S10 · p. 20, Figure S11 · p. 21, Figure S13 · p. 23

G A simplified fly simulation environment

Gym-Fly is a simplified two-dimensional flight task with wind, gravity, noisy actions, and sensory delay. In 30,000 closed-loop episodes, Bio-OFC flies toward the target without significant overshoot, unlike the policy-gradient agent, which overshoots and backtracks.

G A simplified fly simulation environment

  • Gym-Fly simulates two-dimensional flight with independently controlled wing-flapping frequencies, gravity, changing wind, stochastic action noise, and observations delayed by 100 ms.The sensory observations include location, velocity, and wind.Source: G A simplified fly simulation… · p. 25
  • The task tests Bio-OFC beyond its derivation assumptions by using L1 distance cost and uniform rather than Gaussian control noise.Source: G A simplified fly simulation… · p. 25
  • Gravity requires a bias because the zero-state, zero-action point is not fixed; the model adds this by appending a constant component to the estimated state.The authors describe a postsynaptic thresholding mechanism as a biologically plausible implementation.Source: G A simplified fly simulation… · p. 25
  • In the delayed Gym-Fly task, the policy-gradient agent overshoots and backtracks, whereas Bio-OFC flies directly toward the target with no significant overshoot.Bio-OFC was trained in closed loop for 30,000 episodes.Source: G A simplified fly simulation… · p. 25, Figure S16 · p. 26
Loading 2111.06920v1…