PaperlayerView exact revision on arXiv

Source-linked AI summary

Making brain-machine interfaces robust to future neural variability

David Sussillo, Sergey D. Stavisky, Jonathan C. Kao, Stephen I. Ryu, Krishna V. Shenoy

arXiv:1610.05872v1q-bio.NCstat.ML

TL;DR

BMI decoders trained on recent data can become ineffective when neural recording conditions change. This study trained a multiplicative recurrent neural network decoder on varied historical and synthetic conditions, and found it robust where a state-of-the-art Kalman filter decoder failed.

Abstract

from arXiv · show

A major hurdle to clinical translation of brain-machine interfaces (BMIs) is that current decoders, which are trained from a small quantity of recent data, become ineffective when neural recording conditions subsequently change. We tested whether a decoder could be made more robust to future neural variability by training it to handle a variety of recording conditions sampled from months of previously collected data as well as synthetic training data perturbations. We developed a new multiplicative recurrent neural network BMI decoder that successfully learned a large variety of neural-to- kinematic mappings and became more robust with larger training datasets. When tested with a non-human primate preclinical BMI model, this decoder was robust under conditions that disabled a state-of-the-art Kalman filter based decoder. These results validate a new BMI strategy in which accumulated data history is effectively harnessed, and may facilitate reliable daily BMI use by reducing decoder retraining downtime.

RESULTS · Robustness to unexpected loss of the most informative electrodes

Across the results, MRNNs benefited from training on many days of neural data and remained effective when informative electrode inputs were unexpectedly lost, outperforming a same-day FIT-KF in closed-loop tests. Training data augmentation that simulated reduced firing rates contributed to electrode-drop robustness.

An MRNN can leverage large amounts of training data to improve decoder performance

Robustness to unexpected loss of the most informative electrodes

Robustness to naturally occurring recording condition changes

With training data spanning months before a gap, the MRNN was consistently usable under naturally occurring recording changes, whereas FIT-KF decoders were not; offline analyses found that more training data and, to a lesser extent, spike-rate perturbations contributed to MRNN robustness.

Robustness to naturally occurring recording condition changes

  • Tests withheld the most recent several months of recordings, training MRNN on many earlier months and comparing it with FIT Old trained on the latest prior day and FIT Long trained on the same multiday data as MRNN.The intervening months were defined as the training-data gap.Source: Robustness to naturally occurring… · p. 10
  • MRNN acquisition rate was 105% higher than FIT Long (p < 0.01, paired t-test); Monkey L’s MRNN performed less consistently but showed a 32% improvement trend over FIT Long (p = 0.45).The MRNN was the only decoder reliably usable when trained.Source: Robustness to naturally occurring… · p. 11
  • FIT Old failed in 4/6 Monkey R sessions and 6/6 Monkey L sessions when trained on stale data.Failure was defined as the monkey being unable to complete a block using the decoder.Source: Robustness to naturally occurring… · p. 11
  • FIT Long performed better than FIT Old but varied across test days, sometimes failing to acquire targets, while MRNN outperformed it on all but one Monkey L test day.FIT Long was similar to or substantially worse than MRNN on the other Monkey L test days.Source: Robustness to naturally occurring… · p. 11
  • Offline simulations across non-overlapping training and test datasets spanning each animal’s research career showed the same pattern: more prior training days and, to a lesser extent, spike-rate perturbations contributed to MRNN robustness.MRNNs trained with many previous days outperformed FIT Old and FIT Long.Source: Robustness to naturally occurring… · p. 12

High performance BMI using the MRNN decoder

The MRNN outperformed a FIT Sameday decoder in online target acquisition, retained performance under ideal recording conditions, and enabled cursor control after training on months of data. It also supported control on a Random Target Task beyond the task structure that dominated its training data.

High performance BMI using the MRNN decoder

  • MRNN robustness to challenging recording conditions did not reduce performance under more ideal conditions without electrode dropping or stale training data.Source: High performance BMI using the MRNN… · p. 12
  • After training on several months of data through the previous day, the MRNN enabled both monkeys to control the cursor accurately and quickly.Supplementary Movie 1 shows example cursor control.Source: High performance BMI using the MRNN… · p. 12
  • Monkey R acquired targets 7.3% faster with the MRNN (619 ± 324 ms vs. 668 ± 469 ms), and Monkey L 10.8% faster (743 ± 390 ms vs. 833 ± 532 ms); both p < 0.01.The comparison was with a FIT Sameday decoder.Source: High performance BMI using the MRNN… · p. 13
  • Despite concern that training on similar reaches could impair generalization, MRNN control on the Random Target Task achieved 99.4% success for Monkey R and 100% for Monkey L.The task placed targets anywhere in a 20 x 20 cm workspace; mean distance-normalized time to target was 0.068 s/cm for Monkey R and 0.075 s/cm for Monkey L.Source: High performance BMI using the MRNN… · p. 13
  • For context, arm-control success rates on the Random Target Task were 100% for Monkey R and 97.7% for Monkey L, with mean normalized time to target of 0.055 s/cm for each.Source: High performance BMI using the MRNN… · p. 14

DISCUSSION

The discussion presents the MRNN as a fixed decoder trained on accumulated, perturbed data to tolerate recording-condition changes while maintaining performance under favorable conditions, and identifies limits and future tests. It also advocates training computationally powerful nonlinear BMI decoders on large datasets spanning varied conditions, including past closed-loop BMI and routine device-generated data.

  • The MRNN remained usable under stale-data conditions that typically require FIT-KF recalibration, tolerated two recording changes, and slightly outperformed same-day FIT-KF under favorable conditions.The authors report that robustness did not come at the cost of diminished performance under more ideal conditions.Source: DISCUSSION · pp. 15–16
  • Electrode-dropping experiments showed robustness to unexpected loss of high-importance electrodes and to recording changes not encountered in past recordings.The authors describe this as progress on the “errant unit” problem and evidence that training with artificially perturbed neural data can help with unseen changes.Source: DISCUSSION · p. 16
  • The MRNN required multiple hours to train in this study and appears to need more data to reach saturation than conventional methods, although accumulated recordings are its intended resource.The authors did not optimize training speed and suggest iterative updates or faster computing as ways to reduce training time.Source: DISCUSSION · p. 17
  • Robustness came from combining a large training corpus, deliberate data perturbations, and an architecture able to learn diverse data; past data help when they capture mappings similar to current conditions.The authors’ offline and closed-loop results suggest this held for the two monkeys studied.Source: DISCUSSION · p. 17
  • The authors note that whether accumulated data will be relevant to human BMI conditions remains an important question for future validation.They report indications that it may be relevant, but the present robustness results were from two monkeys.Source: DISCUSSION · p. 18
  • A decoder that works across conditions can avoid switching-related downtime, but comparing this fixed-decoder strategy with adaptive decoding, alone or in combination, remains future work.The discussion notes that adaptive decoding requires a period when control is good enough to infer user intentions and update the model.Source: DISCUSSION · pp. 15, 18
  • Random Gaussian firing-rate scaling proved effective, and the authors propose targeting augmentation to specific changes such as electrode loss, array movement, degradation, or cognitive-state interactions.They identify more specific modeling of the changes an experimenter wants to guard against as a future research direction.Source: DISCUSSION · p. 18
  • Training computationally powerful nonlinear BMI decoders on large quantities of data spanning different recording conditions is a general decoder strategy.Source: DISCUSSION · p. 19
  • Neural data and corresponding kinematics from past closed-loop BMI use can serve as training data, even when not collected explicitly for training [4, 10].Source: DISCUSSION · p. 19
  • Other nonlinear decoding algorithms may also benefit, creating opportunities to improve BMI reliability and performance by using the large quantities of data generated during regular device use.Source: DISCUSSION · p. 19

METHODS · Animal model and neural recordings

The study used two rhesus macaques with chronic motor-cortical arrays, pooled M1 and PMd electrodes for decoding, and recorded threshold crossings under a millisecond-timed BMI setup. Neural signal quality was characterized using electrode-tuning and cross-talk measures.

Animal model and neural recordings

  • Two adult male rhesus macaques, ages 8 and 18, had 96-electrode arrays implanted 30 and 74 months before the primary experiments, respectively.Procedures and experiments were approved by Stanford University's Institutional Animal Care and Use Committee.Source: Animal model and neural recordings · p. 20
  • Monkey R had arrays in caudal PMd and M1, while monkey L had an array at the PMd–M1 border.Locations were estimated visually from anatomical landmarks.Source: Animal model and neural recordings · p. 20
  • In the reach behavior studied, responses were qualitatively similar between M1 and PMd, so electrodes were not distinguished, consistent with standard BMI decoding practices [6,8,10,24,38,40,46].Prior reports describe a rostral gradient of increasing preparatory activity rather than stark qualitative differences [52–56].Source: Animal model and neural recordings · p. 20
  • Behavioral control and neural decoding ran on separate PCs using xPC Target, and neural data reached the control system within 5 ± 1 ms.The platform enabled millisecond-timing precision for computations.Source: Animal model and neural recordings · p. 20
  • Spike counts were obtained as threshold crossings using a single negative threshold set to -4.5 times each electrode’s spike-band root mean square.Threshold crossings can contain spikes from one or more nearby neurons; they were used as standard intracortical BMI signals [1,4,6,7,10,15,16,31,38,40].Source: Animal model and neural recordings · pp. 20–21
  • Supplementary Data 1 reports statistics for several measures of electrode tuning and cross-talk to characterize the neural signals available during the study.The paper notes threshold crossings provide roughly comparable population-level velocity decoding to sorted single-unit activity without time-consuming sorting, and may be more stable over time [30,45,57–59].Source: Animal model and neural recordings · p. 21

Behavioral tasks

The behavioral protocol involved virtual-cursor reaching under hand or neural control in Radial 8 and Random Target tasks, with repeated baseline reaches across sessions. It also specified how disengagement guided decoder-testing endpoints and failure recording, and how sessions or additional block-sets were selected or adjusted.

Behavioral tasks

  • The monkeys controlled a virtual cursor with either the contralateral hand or neural activity, with the reaching arm unrestrained during BMI control.Reaches were made to virtual targets in a 2D fronto-parallel plane within a 3D environment.Source: Behavioral tasks · p. 21
  • In the Radial 8 Task, the monkey acquired targets alternating between a center target and one of eight equally spaced peripheral targets on a circle, positioned 8 cm from center.An acceptance window centered on the target was used for 500 ms.Source: Behavioral tasks · pp. 21–22
  • Baseline hand-velocity correlations across sessions averaged 0.90 ± 0.04 for monkey R and 0.91 ± 04 for monkey L.Similarity was calculated by averaging Pearson correlations for the x- and y-velocity vectors; the animals were already well trained, with modest day-to-day variability.Source: Behavioral tasks · pp. 22–23
  • Radial 8 performance was measured as targets acquired per minute and time to target, with time comparisons restricted to conditions without significantly different success rates.Targets per minute reflects acquired targets over task duration, so failures reduce the metric; time to target was measured on successful peripheral-target trials.Source: Behavioral tasks · p. 23
  • In the Random Target Task, each target appeared at a random location within a 20 cm × 20 cm region centered in a 40 × 30 cm workspace; performance was measured by success rate and normalized time to target.Normalized time to target was calculated for successful trials following another successful trial.Source: Behavioral tasks · p. 24

Decoder comparison experiment design

  • Offline MRNN and FIT-KF comparisons used held-out test data and matched training data, while online decoder comparisons used interleaved blocks with about 20 transition trials.Decoder order was randomized across days for electrode-dropping experiments; stale-data decoder order was manually adjusted to alternate more and less frustrating conditions.Source: Decoder comparison experiment design · pp. 24–25

Behavioral tasks

  • Blocks stopped when success fell below 50% after at least 10 trials, a threshold chosen because monkeys soon stopped performing and often refused to re-engage.When a decoder could not complete a block, it was recorded as failed and scored as 0 targets per minute for that session.Source: Decoder comparison experiment design · p. 26
  • In electrode-dropping sessions, both FIT-KF Sameday and MRNN were declared failed at a drop level if neither could complete a block; unusable decoders were not retested.This exception avoided further testing that would have unduly frustrated the animal.Source: Decoder comparison experiment design · p. 26
  • The study used two monkeys with no exclusions; predetermined sessions were selected for offline analyses, while electrode-drop sessions were adjusted to explore decoder robustness across drop conditions.For stale-training experiments, two gaps with three test days each were established beforehand; electrode-drop sessions continued until the animal became satiated or disengaged.Source: Decoder comparison experiment design · pp. 26–27
  • No formal effect-size calculations set sample sizes, but decoder comparisons included many tens of thousands of trials; secondary online experiments used 1–3 sessions per animal.Secondary sessions supported offline analyses or tested MRNN acquisition of other target locations, conserving experimental time.Source: Decoder comparison experiment design · p. 27

Neural decoding using a Multiplicative Recurrent Neural Network (MRNN)

The MRNN decoder converts observed spike counts into cursor position and velocity by reading kinematic outputs from a recurrent network’s state. Its input multiplicatively interacts with the hidden state by parameterizing the recurrent weights, allowing the network to select a decoder suited to current neural statistics.

Neural decoding using a Multiplicative Recurrent Neural Network (MRNN)

  • The decoder maps observed electrode spike counts to cursor position and velocity by reading training kinematics from the recurrent network’s state.The network is trained on time series of neural inputs so that the corresponding kinematic outputs can be read from its state.Source: Neural decoding using a Multiplicative… · pp. 27–28
  • MRNN inputs parameterize recurrent weights, creating multiplicative interactions with the hidden state that can select a decoder suited to current data statistics.Unlike a standard RNN, where input acts as an additive time-dependent bias, the MRNN uses input to parameterize its recurrent weight matrix.Source: Neural decoding using a Multiplicative… · p. 28
  • The input-dependent recurrent-weight tensor is factorized into F factors to make computation tractable, with F controlling interaction complexity.A full tensor is impractical for continuous-valued inputs or discrete inputs with too many possible values.Source: Neural decoding using a Multiplicative… · pp. 28–29

MRNN training

MRNN training combined data from multiple recording days with repeatedly perturbed spike inputs to train neural-to-kinematic mappings. The networks were trained offline, with held-out cross-validation used to select a saved model.

MRNN training

  • Network sizes were N=100 for monkey R and N=50 for monkey L, with F=N and E=192 and E=96, respectively.Monkey R had two multielectrode arrays and monkey L had one.Source: MRNN training · pp. 29–30
  • Five consecutive reaching trials formed each MRNN training trial: two seeded the hidden state and the next three supplied learning data.Training windows advanced by one actual trial, excluding the first two trials of each recording day from learning.Source: MRNN training · p. 30
  • Offline training minimized average squared kinematic error using Hessian-Free optimization, and the saved network with the lowest held-out cross-validation error was selected.Optimization ran for 200 steps, with network snapshots saved every 10 steps.Source: MRNN training · pp. 30–31
  • Separate MRNNs learned two-dimensional hand position and hand velocity outputs, with velocity calculated from position using central differences.Source: MRNN training · p. 31
  • Although each decoder was retrained when data were added in this study, existing MRNN weights could initialize optimization on an expanded corpus for faster convergence.Retraining from scratch was used to verify consistent convergence to a high-quality decoder.Source: MRNN training · p. 31

Improving the MRNN by training with many datasets and intentionally perturbed inputs

  • Training minibatches sampled trials from multiple recording days so each minibatch represented the input distributions of all training days.The datasets spanned months of previous recordings.Source: Improving the MRNN by training with… · p. 31
  • Input spike trains were perturbed anew during optimization by adding or removing spikes to model global firing-rate modulation and electrode-specific changes.Perturbations were applied only during training; closed-loop control used true spike counts.Source: Improving the MRNN by training with… · p. 32
  • When perturbing an electrode’s spike count, spikes were added to random bins or removed from bins that contained spikes, with resampling to preserve the approximate average count.If the perturbed count was unchanged, the trial was left unmodified.Source: Improving the MRNN by training with… · p. 32

Controlling a BMI cursor with MRNN network output

The MRNN provided closed-loop cursor control by blending position and velocity outputs. Its parameters remained fixed during use, so robustness to input changes was inherent rather than adaptive.

Controlling a BMI cursor with MRNN network output

  • In closed-loop control, paired MRNNs consumed binned spike counts and blended position and velocity outputs to estimate and update cursor position.Decoded position and velocity and the MRNN hidden state were initialized to zero.Source: Controlling a BMI cursor with MRNN… · p. 33
  • With β=0.99, online decoding was almost entirely velocity-based, with a small position contribution to stabilize accumulated drift.For offline accuracy comparisons with the velocity-only FIT-KF decoder, β was set to 1.Source: Controlling a BMI cursor with MRNN… · p. 33
  • MRNN parameters remained fixed during closed-loop use, preventing it from learning changed neural-to-kinematic mappings online; its robustness was inherent rather than adaptive.The standard procedure retrained the MRNN with additional data after each experimental session.Source: Controlling a BMI cursor with MRNN… · p. 34

Neural decoding using a Feedback Intention Trained Kalman Filter (FIT-KF)

The FIT-KF estimates cursor kinematics from binned neural observations using a Kalman filter trained on hand-reach data. It adds target-directed intention and perfect visual-feedback assumptions to its training and inference.

Neural decoding using a Feedback Intention Trained Kalman Filter (FIT-KF)

Mutual information for determining electrode dropping order

Electrode loss was tested by ranking electrodes with a decoder-independent measure of how informative their binned spike counts were about reach direction, then dropping them from highest to lowest mutual information.

Mutual information for determining electrode dropping order

Principal angles of neural subspaces analysis

The analysis used minimum principal angles between recording-day neural subspaces as a scalar measure of similarity in reaching-related activity.

Principal angles of neural subspaces analysis

  • The analysis compared recording days using the minimum principal angle between neural subspaces defined by leading PCs of reaching-related neural coactivations.The subspaces represent patterns of covariance across electrodes during arm reaching.Source: Principal angles of neural subspaces… · p. 36
  • Input data comprised arm-control Radial 8 reaching activity from ~200 trials, represented across electrodes in non-overlapping 20-ms bins and centered by each electrode’s across-days mean firing rate.Source: Principal angles of neural subspaces… · p. 36
  • The subspaces were based on leading PCs from covariance-matrix eigendecomposition; Supplementary Fig. 1 used K=10, with qualitatively similar appearance for K=2–30.Source: Principal angles of neural subspaces… · pp. 36–37

FIGURES (with legends below) · Supplementary Information · Supplementary Video 1. Example closed-loop performance of the MRNN

The figures and supplementary analyses show that MRNN decoders trained on accumulated data can handle future recording conditions, including stale neural inputs and electrode loss. They also examine recurring neural recording conditions and training perturbations that improve robustness.

FIGURES (with legends below)

  • Across 154 monkey R and 250 monkey L recording days, the MRNN outperformed FIT Sameday and FIT Long on every day’s dataset.Using more previously collected data also improved MRNN accuracy, especially on chronologically distant test days.Source: Figure 2 · p. 44, FIGURES (with legends below) · pp. 44–45
  • With up to 10 informative electrodes silenced, the MRNN significantly outperformed FIT Sameday in some electrode-loss conditions across sessions.The tests used electrode-dropping orders based on mutual information with target direction.Source: Figure 3 · p. 46, FIGURES (with legends below) · p. 46
  • Only the stale-trained MRNN remained usable on every test day when recent months of data were withheld.The comparison included FIT Long, FIT Old, and FIT Sameday decoders across six experiment days for each monkey.Source: Figure 4 · p. 47, FIGURES (with legends below) · p. 47

Supplementary Information

  • Across months of recordings, similar neural recording conditions sometimes recurred on both nearby and more distant days.The analysis included 71 sessions over 658 days for monkey R and 125 sessions over 1003 days for monkey L.Source: Figure 1 · p. 49, Supplementary Information · p. 50
  • Spike-rate perturbation augmentation improved MRNN robustness to electrode loss and naturally occurring recording changes.Offline analyses also found improved robustness from perturbations, while larger training datasets improved performance on future test data; MRNN Big with perturbations achieved the best performance across individual gaps, with stated exceptions for monkey L.Source: Figure 3 · p. 51, Figure 4 · p. 52, Figure 6 · p. 54
  • Both monkeys acquired Random Target Task targets across a broad workspace, including locations that differed from those dominating training data.Monkey R had one failed trial; monkey L’s data came from one session.Source: Figure 7 · p. 55

Supplementary Video 1. Example closed-loop performance of the MRNN

  • The supplementary videos show 90 seconds of MRNN Radial 8 control and a 60-second comparison with FIT Sameday after the same three electrodes were dropped.The MRNN was trained on 125 sessions through the previous day; FIT Sameday used data collected earlier that day.Source: Supplementary Video 1. Example… · p. 56
Loading 1610.05872v1…