Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity

1 Introduction

• Large, dynamic recordings from diverse brain areas can stymie analytical approaches designed for one or a handful of neurons.The paper describes these recordings as increasingly available through advances in neural interfacing.Source: §1 · p. 1
• Latent variable models characterize population covariation to reveal internal state and support summarizing, visualizing, and relating activity to behavior.The paper also notes their use in studying dynamic mechanisms of population-level computations.Source: §1 · p. 1
• NLB addresses the lack of standardized datasets and metrics with benchmark suites offering curated data, standardized APIs, and example codepacks.The stated goal is to support standardized quantitative evaluation of latent variable models on neural data.Source: §1 · p. 2

1.1 Scientific Motivation and Evaluation Philosophy

• Neural population recordings provide partial views of intermediate representations, but limited sampling, unknown connectivity, and noisy responses complicate their interpretation.These challenges make the task statistical and well-suited to latent variable approaches, as the paper frames it.Source: §1.1 · p. 2
• Because latent variable models can make different assumptions and have varied structures, co-smoothing evaluates their ability to describe observed neuronal activity without prescribing model form.The paper presents this unsupervised evaluation as a way to retain flexibility while avoiding intricate comparisons among differing model structures.Source: §1.1 · p. 3

1.2 Related Work

• Nonstandard datasets and metrics make it difficult to compare models or track progress; NLB differs from related benchmarks by evaluating latent-variable descriptions of neural activity unsupervised.The paper also argues that supervised latent evaluation may be limiting when external variables are unreliable or incomplete descriptions of activity.Source: §1.2 · p. 3

2 NLB ’21: Datasets

• NLB ’21 spans motor, sensory, and cognitive activity, and the paper reports no earlier benchmark for neural-data latent-variable evaluation.The four area/task combinations include motor-cortical Maze and random-target reaching, somatosensory Area 2 movements, and the DMFC Ready-Set-Go task.Source: §2 · p. 3, Figure 1 · p. 4
• All datasets contain spike-sorted electrophysiological measurements from intracortical microelectrode arrays and are available in the NWB standard.The authors treat spike sorting as distinct from extracting latent structure and use selected sessions from larger datasets.Source: §2 · p. 4
• MC_Maze combines repeated, stereotyped reaches across many conditions with high trial counts, enabling trial averaging and separation of preparation from execution.The full dataset has 2,869 trials and 182 neurons; scaled versions use 500, 250, or 100 training trials and 100 test trials each.Source: §2 · pp. 4–5
• MC_RTT provides continuous, variable-length reaches with few repetitions and random evaluation snippets, challenging trial averaging and autonomous-dynamics assumptions.The dataset covers 15 minutes of reaching, split into 1,351 600 ms trials, with 130 neurons and hand kinematics.Source: §2 · p. 5
• Area2_Bump records somatosensory activity during reaches, with unexpected arm perturbations in a random half of trials and associated kinematics and perturbation information.It contains 462 trials and 65 neurons; the text highlights inferring perturbation inputs and robustness to low neuron counts as likely modeling challenges.Source: §2 · p. 5
• DMFC_RSG captures cognitive interval reproduction, where animals infer an interval from sensory cues and reproduce it through eye or hand movements.The dataset has 40 task conditions, 1,289 trials, and 54 neurons; likely modeling challenges include balancing input- and internally-driven dynamics and robustness to low neuron counts.Source: §2 · p. 5

3.1 Evaluation strategy and pipeline

• NLB ’21 standardizes multi-axis LVM evaluation while leaving training and inference unrestricted; submitted firing-rate predictions are scored against private test data.Only held-in test data are provided to users; withheld test data remain on EvalAI servers.Source: §3 · p. 5, §3.1 · p. 6, Figure 2 · p. 6

3.2 Primary evaluation metric: Co-smoothing

• Co-smoothing scores held-out-neuron log-likelihood in bits per spike, comparing inferred time-varying activity with a mean-rate baseline.It assumes held-in neurons supplied at test time support inference of latent variables underlying held-out neurons.Source: §3.2 · pp. 6–7

3.3 Secondary metrics

• Behavioral decoding is secondary: linear ridge R2 predicts hand velocity in three datasets, while DMFC_RSG uses neural-speed correlation with response time.Source: §3.3 · p. 7
• PSTH matching compares trial-averaged predicted rates with true PSTHs using R2, but omits single-trial variability and is unsuitable for MC_RTT.PSTH matching is also limited to datasets with suitable trial structure.Source: §3.3 · p. 7
• Forward prediction tests responses of all neurons at future time points, but is best suited to data modeled by autonomous dynamics, such as MC_Maze.Unpredictable inputs make its assumption that future activity follows from prior activity problematic.Source: §3.3 · p. 7
• Across MC_Maze metrics, baseline rank order is consistent, and NDT and AutoLFADS improve on less expressive baselines.Source: Table 1 · p. 8

3.4 Baselines to seed the benchmark

• Five seed baselines range from smoothed spikes and GPFA to switching linear systems (SLDS), recurrent variational autoencoding (AutoLFADS), and transformer generation (NDT).Source: §3.4 · p. 8

4.1 MC_Maze as Neuroscience MNIST

• On MC_Maze, baseline model rankings are largely consistent across metrics, with AutoLFADS and NDT stronger across the board.This consistency supports co-smoothing in a well-structured dataset and motivates its use on more challenging datasets.Source: §4.1 · p. 8
• Smoothing AutoLFADS rates improves behavioral decoding but decreases co-smoothing, while GPFA underperforms spike smoothing on co-smoothing yet outperforms it on behavioral decoding.The authors suggest GPFA's coarse, slowly evolving inferred rates may explain the contrast, while noting that further tuning could improve its performance.Source: §4.1 · p. 8

4.2 Results across datasets

• Across datasets, deep networks tend to perform best, but MC_RTT shows a smaller linear–deep gap while only deep models beat spike smoothing on DMFC_RSG.Absolute co-smoothing scores are not easily compared across datasets.Source: §4.2 · pp. 9–10, Table 2

4.1 MC_Maze as Neuroscience MNIST

• Forward prediction omits spike smoothing and GPFA, while NDT and AutoLFADS outperform the firing-rate null and infer MC_Maze's predictable autonomous dynamics.SLDS scores are negative and underperform the simple average-firing-rate estimate.Source: §4.1 · p. 8, DMFC_RSG · p. 9

4.2 Results across datasets

• Matched MC_Maze subsets support more direct dataset-size comparisons; AutoLFADS is more stable on smaller datasets, while NDT lags slightly on MC_Maze-S.Across datasets, neuron counts and firing rates vary, making direct score comparisons difficult.Source: §4.2 · p. 10

5 Discussion

• NLB’21 is an initial step toward standardized evaluation, and the authors hope it will catalyze broader LVM adoption in systems neuroscience.Source: §5 · p. 10
• The model-agnostic framework compares 5 approaches across 7 real neuroscientific datasets, making this the broadest LVM comparison yet attempted, to the authors’ knowledge.The platform is intended to support further comparison through community-driven contributions.Source: §5 · p. 10
• The initial evaluation identifies deep networks as powerful tools, while allowing alternate training strategies and imposing no computational or memory restrictions.The authors encourage submissions to report computational and memory requirements.Source: §5 · p. 10
• The first suite anticipates extensions to multi-region interactions, alternate recording modalities, transfer learning, and robust state estimation under recording instability.The authors identify interpretability measures as an important future direction because interpretability is key to an LVM’s ability to drive scientific progress.Source: §5 · p. 10
• The authors hope standardized benchmarking can reduce redundant animal data collection and help researchers get more use from existing data.Source: §5 · p. 10
• Potential LVM applications to brain-machine interfaces and therapeutic devices raise questions about identity, normality, authority, responsibility, privacy, and justice.The passage describes these human impacts as subjects of ongoing study.Source: §5 · p. 10

A.1.1 Maze co-smoothing and decoding

• Around 0.9 decoding accuracy was an apparent upper bound, yet LFADS and NDT fits showed a wide range of co-smoothing scores there.The figure caption describes good behavior decoding across a wide range of fits to neural data.Source: §A.1.1 · p. 18, Figure 4 · p. 18
• The analysis evaluated 20 LFADS and 20 NDT models with random parameters on MC_Maze.Source: §A.1.1 · p. 18
• The wide co-smoothing range at similar decoding accuracy indicates that co-smoothing captures an additional model dimension not captured by behavioral decoding.Source: §A.1.1 · p. 18

A.1.2 Co-smoothing bootstrap analysis

• The reliability analysis compares model performances pairwise on 100 bootstrap samples of the test set.Source: §A.1.2 · p. 18
• Most paired model comparisons fall on one side of the diagonal across bootstrap samples, indicating robust model rankings across evaluation datasets.Each point compares two models on the same bootstrap sample.Source: Figure 5 · p. 22

A.2 The rapid growth of neural LVMs

• Related neural LVM works in ML venues use a wide variety of datasets and evaluation strategies.The passage refers to Table 3 as highlighting these works and strategies.Source: §A.2 · p. 22
• Even when studies list the same dataset or metric, implementations rarely align because they vary in dataset subsets, metric normalization baselines, and model inputs and outputs.Source: §A.2 · p. 22
• The documented variation provides a semi-quantitative account of the need for NLB.Source: §A.2 · p. 22

• Datasets are converted to Neurodata Without Borders (NWB), a standardized format for neurophysiological data.Source: §A.3.1 · p. 22
• NWB datasets can be read with Python and Matlab APIs or accessed as HDF5 files from any language with an HDF5 package.Source: §A.3.1 · p. 22
• The benchmark’s nlb_tools package facilitates reading converted NWB files and extracting relevant data.Source: §A.3.1 · p. 22
• Table 3 surveys neural LVMs published in ML venues over 15 years; checks under metric columns indicate the data used to evaluate inference.Source: Table 3 · p. 23

A.3.2 Data hosting and licensing

• The datasets are hosted on DANDI, which generates structured metadata and persistent identifiers, and are distributed under a Creative Commons Attribution 4.0 International license.The authors state that they bear responsibility in case of rights violations.Source: §A.3.2 · p. 24

A.3.3 Dataset documentation

• The datasets cover movement-preparation dynamics, naturalistic self-paced reaching, predictable and unexpected proprioceptive perturbations, and cognitive timing without a clear moment-by-moment behavioral correlate.Spike sorting can assign the same neural activity to separate units across electrodes, motivating removal of units based on cross-correlations.Source: §A.3.3 · pp. 24, 25, 27, Area2_Bump · pp. 29, 31

A.4 Baselines

• Five baselines of varying complexity were applied to all seven datasets, with three random-seed runs and mean scores ± standard error reported, except for spike smoothing.Spike smoothing was run once because it consistently converged to the same solution regardless of random initialization.Source: §A.4 · p. 33

Spike smoothing

• Spike smoothing fits a Poisson GLM from log-transformed smoothed spikes to held-out activity, with kernel width and regularization selected by grid search and five-fold training cross-validation.The selected parameters are refit on the full training set for test predictions.Source: Spike smoothing · p. 33

GPFA

• GPFA transforms held-in test spikes into latent factors, then uses linear regression and a Poisson GLM to predict held-out spikes, with dimensionality and penalties selected by grid search and three-fold cross-validation.The best settings are refit on the entire training set before test predictions.Source: GPFA · p. 33

SLDS

• The SLDS implementation used a modified ssm package, training on all neurons for required and forward-prediction steps and approximating test posteriors with masks for missing held-out data.Rate predictions were generated by smoothing observations from both training and test data.Source: SLDS · pp. 33–34

Spike smoothing

• SLDS parameters were selected by random search and follow-up training, but the authors noted that time constraints may have left room to improve performance.They searched 20 parameter combinations on trial subsamples, then trained the three best combinations on full training data for 50 and 100 iterations and selected the best of six models.Source: SLDS · p. 34

AutoLFADS

• AutoLFADS used held-in activity as encoder input and predicted held-in and held-out rates, while NDT treated held-out neurons and timesteps as masked prediction targets.AutoLFADS used Population Based Training; its modified code was not publicly released at the time, while NDT code was publicly available.Source: AutoLFADS · pp. 34–35

A.5.1 Trial alignment

• MC_Maze windows span 250 ms before to 450 ms after movement onset, capturing preparation and reach; Area2_Bump spans 100 ms before to 500 ms after movement onset (active) or bump response (passive), capturing most corrective movement or reach.MC_RTT uses 600 ms segments, chosen as a fairly typical trial length; comparable or shorter segments are essential for real-time decoding. DMFC_RSG is aligned from 1500 ms before to Go with up to 300 ms of jitter to include the production epoch; this production-only alignment prevents segments from containing data from more than one trial.Source: §A.5.1 · pp. 35–36

A.5.2 Decoding lag

• Decoding lags were selected by five-fold cross-validation, choosing the lag with the highest mean result across five baseline methods; Area2_Bump’s optimal lag was -20 ms.The negative lag reflects the sensory-area recording, where behavior is expected to precede the corresponding neural activity.Source: §A.5.2 · p. 36

A.5.3 PSTH kernel width

• PSTH kernel widths were selected by leave-one-out Poisson-likelihood evaluation; the chosen widths were 70 ms for MC_Maze and DMFC_RSG and 40 ms for Area2_Bump.MC_Maze-L’s 50-ms width was used instead of its 40-ms optimum to match MC_Maze-M and MC_Maze-S.Source: §A.5.3 · p. 36
• MC_Maze-L, MC_Maze-M, and MC_Maze-S each used a 50-ms PSTH kernel width.The 50-ms width for MC_Maze-L was chosen to facilitate comparison with MC_Maze-M and MC_Maze-S, whose optimal widths were also 50 ms.Source: §A.5.3 · p. 36

A.6 DMFC_RSG Changes After Initial Release

• DMFC_RSG evaluation changes addressed misleading PSTH and correlation scores, and a neuron dominating neural-speed estimates led the authors to replace the dataset with one from another session.They changed PSTH evaluation to average model predictions within each condition, jittered trial-alignment windows to prevent trivial correlations, and replaced the dataset after finding one neuron alone could drive excellent correlation scores.Source: §A.6 · pp. 36–37