PaperlayerView exact revision on arXiv

Source-linked AI summary

Making brain-machine interfaces robust to future neural variability

David Sussillo, Sergey D. Stavisky, Jonathan C. Kao, Stephen I. Ryu, Krishna V. Shenoy

arXiv:1610.05872v1q-bio.NCstat.ML

TL;DR

Brain-machine interface decoders can fail when neural recording conditions change, limiting reliable use. This paper trains a multiplicative recurrent neural network on varied historical and synthetic conditions and finds robustness to clinically relevant changes, outperforming a same-day-trained Kalman filter without sacrificing peak performance.

  • Problem

    Brain-machine interface decoders can fail after neural recording conditions change, motivating methods that remain effective under future variability.

  • Method

    The study develops a multiplicative recurrent neural network decoder trained on varied recording conditions from accumulated data and synthetic perturbations.

  • Results

    The MRNN was robust to two types of recording-condition changes, outperformed a same-day-trained state-of-the-art FIT-KF decoder, and preserved peak performance.

  • Takeaways & Limitations

    The findings support harnessing accumulated training history to improve BMI robustness across clinically relevant recording-condition changes.

  • Takeaways & Limitations

    Large-dataset nonlinear decoders may overtrain to the training task structure and fail to generalize.

Abstract

from arXiv · show

A major hurdle to clinical translation of brain-machine interfaces (BMIs) is that current decoders, which are trained from a small quantity of recent data, become ineffective when neural recording conditions subsequently change. We tested whether a decoder could be made more robust to future neural variability by training it to handle a variety of recording conditions sampled from months of previously collected data as well as synthetic training data perturbations. We developed a new multiplicative recurrent neural network BMI decoder that successfully learned a large variety of neural-to- kinematic mappings and became more robust with larger training datasets. When tested with a non-human primate preclinical BMI model, this decoder was robust under conditions that disabled a state-of-the-art Kalman filter based decoder. These results validate a new BMI strategy in which accumulated data history is effectively harnessed, and may facilitate reliable daily BMI use by reducing decoder retraining downtime.

RESULTS · Robustness to unexpected loss of the most informative electrodes

The MRNN remained effective during closed-loop cursor control after unexpected loss of highly informative electrodes, whereas FIT-KF performance deteriorated sharply. Robustness depended on the MRNN architecture and neural-activity perturbation augmentation, while larger datasets improved broad recording-condition generalization but not electrode-dropping robustness alone.

  • An MRNN can leverage large amounts of training data to improve decoder performance: r2 = 0.81 ± 0.04 for Monkey R and r2 = 0.84 ± 0.03 for Monkey L were achieved by MRNNs trained across 154 and 250 days, respectively.These datasets spanned 22 and 34 months, and the MRNN outperformed FIT-Sameday and FIT Long on every test day.
  • Robustness to unexpected loss of the most informative electrodes: The challenge simulated sudden electrode failure by zeroing selected informative-electrode firing rates during Radial 8 Task control without retraining or modifying the decoder.The number of dropped electrodes varied systematically to control challenge severity, and the specific zero-rate electrode sets had not appeared in prior training data.
  • Robustness to unexpected loss of the most informative electrodes: The unfamiliar zero-firing condition retained commonality with previously encountered patterns of activity on the remaining electrodes, supporting MRNN generalization to the perturbation.The MRNN had been trained with a large corpus of reaching data collected through the previous day’s session.
  • Robustness to unexpected loss of the most informative electrodes: 52% more targets per minute for Monkey R and 92% more for Monkey L showed that the MRNN outperformed FIT-KF across electrode-dropped conditions.The MRNN suffered only a modest performance loss after losing up to the best 3 electrodes in Monkey R or 5 in Monkey L, while FIT-KF worsened dramatically.
  • Robustness to unexpected loss of the most informative electrodes: Training-data perturbations generated examples with reduced firing rates on varied electrode subsets, broadly preparing the MRNN for firing-rate reductions.The perturbations also included examples with increased firing rates.
  • Robustness to unexpected loss of the most informative electrodes: Closed-loop comparisons confirmed that perturbation augmentation was important, while offline analyses found both the MRNN architecture and augmentation necessary for electrode-dropping robustness.These analyses also suggested that larger training datasets alone did not impart robustness because prior datasets lacked the specific dropped-electrode conditions.

Robustness to naturally occurring recording condition changes

Across naturally occurring recording-condition gaps, the MRNN was the only reliably usable decoder, whereas FIT-KF decoders were inconsistent or failed. Offline simulations reproduced this advantage and implicated larger historical datasets, with data augmentation contributing secondarily.

  • Experimental design: The stale-training experiments withheld the most recent several months, defining the interval before testing as a training-data gap, and compared MRNNs with FIT Old and FIT Long.MRNNs used many months of recordings preceding each gap; FIT Old used the most recent available day, whereas FIT Long used the same multiday dataset as the MRNN.
  • Offline validation: Offline simulations spanning non-overlapping career-period datasets reproduced the closed-loop result: MRNNs trained on many previous days outperformed both FIT Old and FIT Long.These analyses tested datasets separated by gaps and confirmed the same trends as the closed-loop experiments.
  • Robustness components: MRNN robustness increased primarily with more training data, while spike-rate data augmentation provided a smaller additional contribution.The component analysis varied the number of pre-gap days and the presence of training-data spike-rate perturbations.

High performance BMI using the MRNN decoder

The MRNN supported fast, accurate closed-loop cursor control after training on months of prior data. It outperformed the FIT Kalman decoder in online tests while retaining performance under ideal conditions and generalizing to random target locations.

  • High performance BMI using the MRNN decoder: The MRNN remained robust to challenging recording conditions without reducing performance under ideal conditions lacking electrode dropping or stale training data.Both monkeys accurately and quickly controlled the cursor after MRNN training on several months of data through the previous day.
  • High performance BMI using the MRNN decoder: The MRNN trained from many days of recording conditions outperformed the FIT Kalman filter trained from data collected at the session start.This online finding corroborated the offline results presented in Fig. 2c.

DISCUSSION

The MRNN provides a fixed BMI decoder that remains robust across clinically relevant recording-condition changes without sacrificing peak performance. Its success supports training powerful nonlinear decoders on large, varied historical datasets, while training time, data requirements, and human validation remain limitations.

  • DISCUSSION: The fixed MRNN complements adaptive decoding by potentially preserving sufficient control during neural changes for adaptation to recover performance without interrupting prosthesis use.Comparing this strategy with adaptive decoding, alone and in combination, remains future work.
  • DISCUSSION: The experiments modeled recording changes occurring over hours, but whether historical data remain relevant to current human recording conditions requires further validation.The authors note possible greater human variability from array movement, user state, and electromagnetic interference.
  • DISCUSSION: The MRNN remained usable under recording conditions that would typically require recalibration and slightly outperformed a same-day-trained state-of-the-art FIT-KF under favorable conditions.Together, these results indicate robustness without diminished ideal-condition performance.
  • DISCUSSION: The MRNN tolerated unexpected loss of high-importance electrodes and remained relatively robust to unseen recording-condition changes after artificial neural-data perturbation.These findings address the common “errant unit” failure mode and show that robustness can extend beyond previously encountered conditions.
  • DISCUSSION: Training the nonlinear decoder took multiple hours and appears to require more data to saturate performance than conventional same-day-trained methods.The authors view the data requirement as acceptable because the approach exploits accumulated recordings, while faster incremental or hardware-assisted training remains possible.
  • DISCUSSION: Robustness arose from combining a large historical training corpus, deliberate neural-data perturbations, and a computationally powerful architecture able to learn diverse mappings.The decoder can become more robust as accumulated data expands its library of neural-to-kinematic mappings under different conditions.

METHODS · Animal model and neural recordings

The study used adult male rhesus macaques implanted with 96-electrode Utah arrays in motor cortical areas. Neural decoding used threshold crossings collected with millisecond-scale processing and timing.

  • Animal model and neural recordings: Monkey R had arrays in caudal dorsal premotor cortex and primary motor cortex, whereas monkey L had one array at the PMd–M1 border.Electrode locations were estimated visually from anatomical landmarks.
  • Animal model and neural recordings: Because PMd and M1 showed qualitatively similar response properties during point-to-point reaching, decoding did not distinguish between their electrodes.This matched standard BMI decoding practices and the observed motor-cortical response characteristics.
  • Animal model and neural recordings: Spike counts were obtained by applying a single negative threshold set to -4.5 times each electrode’s spike-band root mean square.The decoder used threshold crossings containing spikes from one or more nearby neurons.
  • Animal model and neural recordings: Threshold crossings were used as standard intracortical BMI signals because they provide comparable population-level velocity decoding to sorted units without time-consuming sorting.Prior work also suggests threshold crossings may be more stable over time30,45,57–59.
  • Animal model and neural recordings: Supplementary Data 1 reported statistics for several measures of electrode tuning and cross-talk to characterize neural-signal quality.These measures oriented the reader to the signals available during the study.

Behavioral tasks

Behavioral evaluation used virtual-cursor reaching tasks controlled by hand position or neural activity, primarily the Radial 8 Task and a random-target task. Across sessions, hand-reaching behavior was highly consistent, and decoder comparisons used held-out data and interleaved online blocks.

  • Behavioral consistency: Hand-velocity correlations across sessions were 0.90 ± 0.04 for monkey R and 0.91 ± 04 for monkey L.Similarity was computed from Pearson correlations between concatenated x- and y-velocity vectors from repeated Baseline Block sessions.
  • Random Target Task: The Random Target Task presented targets at random locations within a 20 cm × 20 cm region and measured success rate and distance-normalized time to target.New targets appeared after every trial, and successive target acceptance areas could not overlap.
  • Decoder comparison experiment design: Offline decoder comparisons used held-out test data, while online comparisons interleaved approximately 200-trial blocks of each decoder with about 20 transition trials before measurement.This design enabled fair comparison of MRNN and FIT-KF despite their different parameter counts and allowed decoder order to be controlled across experiments.

Neural decoding using a Multiplicative Recurrent Neural Network (MRNN)

The MRNN decoder converts electrode spike counts into cursor position and velocity by using input-dependent recurrent dynamics. Its multiplicative interactions allow the hidden state to select a decoder suited to current neural-data statistics while continuous-time dynamics control output smoothness.

  • BMI decoding: The decoder transforms each electrode’s observed spike counts into cursor position and velocity outputs, read out from the recurrent network state.The network is trained so its kinematic outputs can be accurately read from that state.
  • Architecture: MRNN inputs directly parameterize the recurrent weight matrix, creating multiplicative interactions between neural inputs and the hidden state.This contrasts with standard RNNs, where inputs act as additive time-dependent biases.
  • Architecture: The hidden state can therefore select an appropriate decoder for the statistics of the current dataset.
  • Training and dynamics: The MRNN uses continuous-time dynamics, with the time constant set on the order of hundreds of milliseconds to support meaningful interactions and smooth outputs.The continuous-time formulation was adapted from the approach suggested in 35.
  • Architecture: To make input-dependent recurrent weights tractable for continuous-valued inputs, the model linearly combines inputs into F factors and factorizes the weight tensor.The factor count F directly controls interaction complexity.

MRNN training

MRNN decoders were trained offline on sequential reaching-trial data to predict hand position and velocity, using Hessian-Free optimization and cross-validation model selection. Robustness training incorporated data from multiple recording days and synthetic spike perturbations applied during optimization.

  • MRNN training: MRNNs used N=100 hidden units for monkey R and N=50 for monkey L, with F=N in both cases.Monkey R had E=192 input channels and monkey L had E=96.
  • MRNN training: Five consecutive reaching trials formed each MRNN training trial; the first two seeded the hidden state, while the next three trained the network.Training data were incrementally shifted across actual trials, using the full recording set except the first two trials of each day.
  • MRNN training: Two independently trained MRNNs predicted normalized hand position and hand velocity, each as a 2-dimensional signal across horizontal and vertical dimensions.Velocities were calculated numerically from positions using central differences.
  • MRNN training: Robustness training sampled trials from many previous recording days in every minibatch and repeatedly perturbed electrode spike counts by adding or removing spikes during optimization.Perturbations modeled global firing-rate modulation and electrode-specific changes, but true spike counts were used during closed-loop BMI control.

Controlling a BMI cursor with MRNN network output

The trained MRNNs were compiled for real-time closed-loop BMI cursor control, combining velocity- and position-based outputs to update the cursor. During use, their parameters remained fixed, so robustness to recording changes was inherent rather than adaptive.

  • Controlling a BMI cursor with MRNN network output: The trained MRNNs ran in a compiled embedded real-time system to provide online closed-loop BMI cursor control.At each decode step, paired MRNNs received binned spike counts and their outputs updated the drawn cursor position.
  • Controlling a BMI cursor with MRNN network output: With β = 0.99, cursor decoding was dominated by velocity with a slight position contribution that stabilized drift.Offline accuracy comparisons instead used β = 1 to compare the MRNN with the velocity-only FIT-KF decoder.
  • Controlling a BMI cursor with MRNN network output: The MRNN was not adaptive during closed-loop use because its parameters were fixed and could not learn new neural-to-kinematic mappings online.Although it was retrained with additional data after experimental sessions, it did not update from newly encountered recording changes during use.
  • Controlling a BMI cursor with MRNN network output: Its robustness to input changes was therefore inherent to the MRNN architecture and training regime rather than produced by online adaptation.Recurrent connections allow previous inputs to affect near-term processing, but the decoder's parameters remain fixed during closed-loop operation.

Neural decoding using a Feedback Intention Trained Kalman Filter (FIT-KF)

The study compared the MRNN with FIT-KF, a Kalman-filter decoder that estimates cursor position and velocity from binned neural spike counts. FIT-KF is trained from hand-reach data and incorporates intention-based kinematic rotation and visual-feedback assumptions [6,40,65].

  • Neural decoding using a Feedback Intention Trained Kalman Filter (FIT-KF): FIT-KF estimates cursor position, velocity, and bias from current binned spike counts and the previous kinematic-state estimate.Its 25-ms observations update the state recursively under a linear dynamical-system model.
  • Neural decoding using a Feedback Intention Trained Kalman Filter (FIT-KF): The decoder’s linear-system parameters are learned under supervision from hand-reach training data [6,65].
  • Neural decoding using a Feedback Intention Trained Kalman Filter (FIT-KF): FIT-KF rotates training kinematics assuming the monkey intends to move directly toward the target [40].
  • Neural decoding using a Feedback Intention Trained Kalman Filter (FIT-KF): FIT-KF assumes perfect visual knowledge of decoded cursor position, setting position-estimate covariance to zero and subtracting position-explainable neural activity [40].

Mutual information for determining electrode dropping order

Electrodes were dropped in descending order of mutual information between binned spike counts and reach direction. This decoder-independent ranking quantified each electrode’s statistical informativeness using entropy differences.

  • Mutual-information ranking: Electrodes were ranked by mutual information between their binned spike counts and reach direction, then dropped from highest to lowest mutual information.The ranking measured how statistically informative each electrode was about reach direction and was independent of the decoder used.
  • Spike-count discretization: Spike counts were discretized into {0,1,2,3,4,5+} bins, with counts ≥5 grouped as 5+, corresponding to 250 Hz in a 20 ms bin.The discretized counts formed the finite set used to compute the entropy and mutual information.
  • Mutual-information ranking: Mutual information was calculated as I(X;Y) = H(Y) − H(Y|X), using each electrode’s entropy and entropy conditioned on reach direction.This formulation quantified the dependence between neural activity and reach direction.

Principal angles of neural subspaces analysis

The analysis quantified similarity between recording days by comparing their neural activity subspaces during arm reaching. Each day’s subspace was defined by the leading principal components of neural coactivations, and day pairs were compared using their minimum principal angle.

  • Metric definition: Day-pair neural similarity was measured as the minimum principal angle between recording-day neural subspaces.This scalar metric captured similarity in reaching-related neural activity patterns between pairs of days.
  • Neural subspaces: Each neural subspace comprised the top K principal components of daily neural coactivations, representing covariance motifs across electrodes during arm reaching.The analysis therefore compared low-dimensional neural activity structure rather than raw activity directly.
  • Data preparation: Daily activity matrices were constructed from approximately 200 Radial 8 Task arm-control trials using non-overlapping 20 ms bins.The task used 8 cm target distances and was performed at the start of almost every session for both monkeys since array implantation.
  • Computation: Principal components were obtained by eigendecomposing each day’s covariance matrix after subtracting each electrode’s across-days mean firing rate.The resulting eigenvectors formed an E×K matrix, with columns ordered by descending eigenvalue.
  • Robustness checks: Supplementary Fig. 1 used K=10 principal components, while varying K from 2 to 30 produced a qualitatively similar pattern.The day-difference metric was computed from the minimum of the K subspace angles between the daily principal-component matrices.

FIGURES (with legends below) · Supplementary Information · Supplementary Video 1. Example closed-loop performance of the MRNN

The figures evaluate MRNN decoding under electrode loss, naturally stale neural inputs, and ideal conditions, while supplementary analyses examine recording variability, perturbation training, and additional closed-loop demonstrations. Together, these materials show how training-data breadth and robustness tests support MRNN BMI performance across changing recording conditions.

  • FIGURES (with legends below): The MRNN used multi-day motor-cortex spike recordings to learn separate internal dynamics for velocity and position readouts from binned spike counts.A monkey performed a target-acquisition task while multiunit spikes were recorded from multielectrode arrays.
  • FIGURES (with legends below): Larger, chronologically earlier training datasets improved MRNN decoding, with training-data recency separated from corpus size in the evaluation design.Except for a few monkey R overlap days, training data preceded the test days.
  • FIGURES (with legends below): With up to 10 informative electrodes removed, MRNN and FIT Sameday closed-loop performance were compared, with significant MRNN advantages marked for specific removal conditions.Performance was measured as targets per minute across experimental sessions; significance used paired t-tests with p < 0.05.
  • FIGURES (with legends below): The stale-input experiment evaluated MRNN, FIT Long, and FIT Old decoders on six test days after withholding several months of recent neural data.The test days spanned 7 days in monkey R and 13 days in monkey L.
  • Supplementary Video 1. Example closed-loop performance of the MRNN: The MRNN also supported ideal-condition cursor control and continuous Radial 8 performance after training on reaching data from 125 recording sessions through the previous day.A supplementary comparison showed alternating-block closed-loop performance after the same three most important electrodes were dropped.
  • Supplementary Information: Supplementary analyses found that chronologically close sessions generally had more similar neural recordings, while occasional similar recordings occurred across distant days.The top 10 covariance eigenvectors captured on average 51% and 46% of single-trial variance for monkeys R and L, respectively.
  • Supplementary Information: Spike-rate perturbation augmentation improved closed-loop MRNN robustness to dropping informative electrodes, and supplementary offline tests separated architecture, dataset size, and augmentation contributions.These analyses addressed both unexpected electrode loss and naturally occurring recording-condition changes.
Loading 1610.05872v1…