PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings

Hanrui Lyu, Baiyuan Chen, Tianshu Tan, Matthew R. Whiteway, Maxwell D. Melin, Ji Xia, Linyang He, Bradly C. Stadie, Anne Churchland, Liam Paninski, Yizi Zhang

arXiv:2610.02864v1cs.LGq-bio.NC

TL;DR

Chronic neural recordings make it difficult to distinguish representational plasticity from recording instability, limiting analysis of how neural representations evolve. NeuroLens predicts future latent states from changing neural populations; across mouse and human data, it improves task-variable decoding and generalizes across sessions. The authors present it as a framework for studying neural representations over long timescales.

  • Problem

    Current analytical tools have difficulty distinguishing representational plasticity from recording instability in chronic neural recordings.

    Source: §1 · p. 1

  • Method

    NeuroLens uses a day-adaptive encoder to map changing neural populations into a common latent space and a shared temporal predictor to predict future latent embeddings.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Across chronic mouse and human intracortical data, NeuroLens improves decoding of decision-making and semantic task variables, generalizes to future sessions, and supports rapid few-shot adaptation to unseen neural populations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    NeuroLens provides a framework for studying task-relevant neural representations over long timescales.

    Source: §6 · p. 10

    The analysis does not disentangle changes due to learning and behavior from intrinsic neural changes or model cross-area interactions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Understanding how neural activity represents higher-order cognition and how these representations evolve over time has long been a central pursuit in neuroscience. However, current analytical tools cannot easily distinguish representational plasticity from recording instability in chronic neural recordings. Here, we introduce NeuroLens (Latent Embeddings of Neural Semantics), a self-supervised model based on the Joint-Embedding Predictive Architecture (JEPA) framework that learns denoised, semantically informative latents from chronic neural recordings. An adaptive encoder maps changing neural populations into a common latent space, while a temporal predictor learns structure that supports prediction of future latent states. By predicting in latent space, NeuroLens captures temporally predictable structure and reduces sensitivity to transient, recording-specific variability. Across chronic intracortical data in mice and humans, the learned representations improve decoding of decision-making and semantic task variables. Multi-day pretraining enables generalization to future sessions, rapid few-shot adaptation to unseen neural populations, and more stable decoding over time than state-of-the-art baselines. Together, these results establish NeuroLens as a new paradigm for studying how neural representations change during learning and over long timescales.

1 INTRODUCTION

NeuroLens addresses the difficulty of separating representational plasticity from recording instability in chronic neural data. It predicts future states in a shared latent space to learn task-relevant representations that generalize across changing populations and recording days.

1 INTRODUCTION

  • Electrode drift, signal degradation, and changes in recorded neurons can obscure learning-related changes, while existing approaches often target specific decoding tasks or compensate for shifts without separating task-relevant structure from recording variability.The paper motivates a general-purpose framework that accommodates changing populations while preserving biological representational changes.Source: §1 · p. 1
  • Reconstruction-based self-supervised models may retain recording-specific details, whereas latent-space prediction emphasizes predictable structure without reproducing every detail of neural activity.Probe drift can change which neurons are recorded even when underlying population dynamics remain stable.Source: §1 · pp. 1–2
  • NeuroLens combines a day-adaptive encoder that maps changing populations into a common latent space with a causal transformer predictor whose parameters are shared across days.The encoder uses cross-attention and infers neuron identities from a few calibration trials; the predictor forecasts future latent embeddings.Source: §1 · p. 2
  • Across mouse and human intracortical data, NeuroLens improves decoding of visual decision-making and semantic task variables and generalizes to future sessions with rapid few-shot adaptation to unseen neural populations.The model also learns temporally consistent representations despite substantial neuron loss and day-to-day drift.Source: §1 · p. 2
  • The framework learns denoised, semantically informative representations through latent prediction and supports analyses of task-relevant neural states as they evolve over time.Its stated contribution is robustness to recording nonstationarity while preserving useful representation dynamics.Source: §1 · p. 2

2 RELATED WORK

Prior neural self-supervised methods often reconstruct activity, while chronic-recording approaches address nonstationarity through alignment, recalibration, or task-oriented transformers. The paper identifies a need for general-purpose representations that remain useful under temporal shifts in neural populations.

Large-scale models for neural analysis

  • Multi-session models include supervised decoding systems and self-supervised methods that mask activity, but supervised representations are tied to predefined targets and many self-supervised objectives focus on reconstruction.Existing multi-session work has focused more on transfer across animals, tasks, or brain regions than on temporal distribution shifts.Source: §2 · p. 2

Self-supervised representation learning for neural activity

  • Observation-space prediction and masked autoencoders model input details, whereas JEPAs predict latent targets to encourage higher-level structure; neural self-supervised models have largely used reconstruction objectives.Latent-space prediction may emphasize dynamics that capture higher-level representations and support generalization.Source: §2 · pp. 2–3

Nonstationarity in chronic neural recordings

  • Chronic-recording methods include manifold alignment, test-time recalibration, and transformer-based approaches, with limits including shared-dynamics assumptions, task-specific pseudo-labels, or supervision for predefined tasks.The cited examples include NoMAD and CEBRA for alignment, CORP and DietCORP for recalibration, and SPINT and POSSM for transformer methods.Source: §2 · p. 3
  • The open challenge is learning general-purpose representations that remain stable as neural population composition changes over long timescales while still supporting accurate decoding.This challenge follows the stated limits of existing approaches to temporal distribution shifts and task specificity.Source: §2 · pp. 2–3

3 DATASETS

The study uses chronic mouse Neuropixels recordings during visual decision learning and chronic human Utah-array recordings during attempted speech. Both datasets provide repeated sessions over extended periods, with neural activity paired to task-related labels or behavioral measures.

IBL chronic Neuropixels data

  • The mouse dataset contains 53 daily sessions across three months, tracking a visual decision task as performance progresses from novice to proficient.It includes trial and spontaneous periods, recordings from hundreds of neurons, and behavioral measures such as choice, wheel speed, whisker motion, and paw and nose speeds.Source: §3 · p. 3

Human chronic Utah array data

  • The human datasets include two participants with ALS-induced anarthria attempting displayed sentences, recorded over 23 sessions in four months and 45 sessions in 20 months.Neural features include threshold-crossing spike counts and spiking-band power; sentence labels include phoneme sequences and semantic embeddings.Source: §3 · p. 3

4 METHODS

NeuroLens maps changing neural populations into a common latent space and uses a shared autoregressive predictor to learn temporally predictable structure. Its architecture, training losses, and adaptation strategies support decoding across days and populations.

4.1 MODEL ARCHITECTURE

  • A day-adaptive cross-attention encoder maps variable-size neural populations into fixed latent embeddings, using calibration-derived neuron signatures and region embeddings; a shared causal predictor forecasts future latents.Each time bin is encoded independently, preventing temporal leakage; calibration-based identities support adaptation to unseen populations.Source: §4 · p. 3, Figure 1 · p. 4, §4.1 · p. 4

4.2 TRAINING OBJECTIVES

  • Training combines mean-squared error between predicted and target latents with SIGReg, which discourages constant representations by encouraging nonzero variance across latent dimensions.The regularizer matches characteristic functions of randomly projected embeddings to an isotropic Gaussian distribution.Source: §4.2 · p. 5

4.3 GENERALIZATION TO UNSEEN DATA AFTER PRETRAINING

  • After multi-day pretraining, gradient-free adaptation uses calibration trials to infer unseen-neuron identities without parameter updates, while gradient-based adaptation updates read-in layers and full fine-tuning updates the entire model.For gradient-based adaptation, the cross-attention layers, region embeddings, and autoregressive predictor remain frozen; full fine-tuning is described for settings with sufficient new-session data.Source: §4.3 · pp. 5–6
  • NeuroLens improves sentence-embedding decoding over causal NDT with comparable phoneme decoding, and improves choice decoding with comparable movement decoding in mouse recordings.The figure also reports lower latent effective rank and clearer choice and semantic separation than causal NDT.Source: Figure 2 · p. 6

5 RESULTS

NeuroLens improves decoding of task variables and generalizes across held-out days in mouse and human recordings. Its latents remain more temporally consistent and dimensionally stable than comparison representations despite chronic recording changes.

5.1 HIGH-LEVEL VS. LOW-LEVEL NEURAL DECODING

  • NeuroLens improves choice decoding and matches movement-related R2 versus causal NDT, with larger reported gains for choice (6%) and sentence embeddings (38%) than movement (2%) and phonemes (4-16%).The authors interpret the pattern as evidence of richer global, semantic structure in the learned representations.Source: §5.1 · p. 7

5.2 GENERALIZATION ACROSS TIME

  • Across 14 held-out mouse days and 5 (T12) and 9 (T15) human days, NeuroLens outperforms baselines in choice, movement, phoneme, and sentence-embedding decoding.Gradient-free adaptation also beats several baselines using only a handful of unlabeled calibration trials and no gradient updates.Source: Figure 3 · p. 8, §5.2 · p. 9

5.3 TASK-RELATED REPRESENTATION DRIFT VS. RECORDING NONSTATIONARITY

  • As recording days grow farther apart, raw-spike representations become more dissimilar while NeuroLens latents remain more similar.The comparison uses trial-by-trial representational similarity across day pairs.Source: §5.3 · p. 9
  • As recorded neuron counts decline, causal NDT’s effective rank decreases substantially, whereas NeuroLens’s rank remains stable across days in both datasets.Figure 4B presents the effective-rank comparison.Source: §5.3 · p. 9
  • The results support stable, compact representations that separate task-related changes from recording nonstationarity for studying learning over long timescales.Source: §5.3 · p. 9

6 DISCUSSION

The discussion presents NeuroLens as a self-supervised model for chronic recordings and identifies unresolved sources of representational change. Appendices describe calibration-based neuron signatures and day-specific channel adaptation.

6 DISCUSSION

  • The study does not disentangle learning- and behavior-related changes from intrinsic neural changes, or model cross-area interactions and region-specific representation drift.The authors identify joint behavioral modeling and cross-area analyses as future directions.Source: §6 · p. 10

A CALIBRATION-BASED NEURON IDENTIFICATION

  • Neuron signatures combine calibration-derived firing-rate statistics, autocorrelograms, and lagged population-coupling features to identify neurons across recording days.The coupling feature correlates each neuron with the activity of the remaining population.Source: A CALIBRATION-BASED NEURON… · p. 13

B CALIBRATION-BASED CHANNEL DRIFT ADAPTATION

  • For channel drift adaptation, normalized calibration statistics are mapped to channel embeddings, averaged by input modality, and projected into an order-independent day context vector.A shared hypernetwork uses that vector to generate day-specific FiLM parameters without updating model weights on an unseen day.Source: B CALIBRATION-BASED CHANNEL DRIFT… · pp. 13–14

C.1 IBL CHRONIC NEUROPIXELS DATA

The experiments use chronic mouse Neuropixels recordings and human Utah-array recordings. Neural inputs, behavioral targets, and language targets are prepared differently for the two datasets.

C.1 IBL CHRONIC NEUROPIXELS DATA

  • The IBL dataset comprises three months of mouse neural and video recordings, with spike counts binned at approximately 60 Hz and trial and non-trial windows analyzed.The mouse performed a visual left-versus-right decision task; video-derived behavior included wheel, whisker, paw, and nose measures.Source: §C.1 · p. 14

C.2 HUMAN CHRONIC UTAH ARRAY DATA.

  • Human T12 and T15 inputs use electrode-level threshold-crossing counts and spike-band power, smoothed and combined into 100 ms tokens for variable-duration trials.T12 uses 128 electrodes in area 6v; T15 uses 256 electrodes in precentral gyrus.Source: §C.2 · pp. 14–15
  • Human decoding targets include phoneme sequences and 384-dimensional sentence embeddings derived from a pretrained sentence encoder.T12 phoneme labels are generated from text; T15 uses the dataset’s supplied phoneme sequences.Source: §C.2 · p. 15

D DATA SPLITS

The data-split design distinguishes same-day representation evaluation from chronological testing on later recording days. For temporal generalization, later days are held out and calibration data are separated from evaluation data.

D DATA SPLITS

  • The same-day setting draws training and test data from the same recording days, while the temporal-generalization setting trains on earlier days and tests on later held-out days.Source: D DATA SPLITS · p. 15
  • For IBL temporal generalization, 39 days are used for multi-day pretraining and the subsequent 14 days for evaluation, with each held-out day split into adaptation, validation, and test portions.The portions are 20%, 10%, and 70%, respectively.Source: D DATA SPLITS · p. 15
  • For human temporal generalization, the earliest 80% of days train the models and the final 20% are held out; held-out-day calibration uses the first 20% of trials.T12 uses 18 training days and 5 test days; T15 uses 36 training days and 9 test days.Source: D DATA SPLITS · p. 15

E METRICS

The section defines metrics for latent dimensionality, semantic decoding, and representational similarity across days. These measures quantify both representation quality and changes in temporal geometry.

E.1 EFFECTIVE RANK OF NEURAL LATENTS

  • Effective rank is the exponential of the Shannon entropy of the normalized eigenvalue spectrum of within-trial covariance, ranging from 1 for collapse to D for equal variance across dimensions.The same number of trials were sampled per day so effective-rank differences were not driven by trial count.Source: §E.1 · pp. 15–16

E.3 TOP-5 SENTENCE RETRIEVAL ACCURACY

  • Top-5 sentence retrieval accuracy counts a trial as correct when the ground-truth sentence is among the five most similar predictions in a candidate pool.On T12 and T15, the target is a 384-dimensional sentence embedding; accuracy is averaged across held-out days.Source: §E.3 · p. 16

E.4 REPRESENTATION SIMILARITY ANALYSIS

  • Representational similarity analysis compares representations through pairwise dissimilarity matrices and is rotation-invariant, allowing comparison across models or days without a shared basis.The matrices encode dissimilarity between time bins within a trial.Source: §E.4 · p. 16
  • Cross-day dissimilarity compares temporal RDMs using one minus their Spearman correlation, with zero indicating identical geometries and larger values indicating greater divergence.Within-day split-half estimates correct for sampling variability, and corrected dissimilarity is visualized against day lag as a model-free characterization of drift.Source: §E.4 · pp. 16–17

F PRETRAINING

Multi-day pretraining is compared with training on the target session alone, and the decoding procedures for phoneme and sentence targets are specified. Pretraining improves movement decoding across tested sessions, while choice decoding favors pretraining on most sessions.

F PRETRAINING

  • The target-session-only baseline was trained from scratch with the same target-session split, architecture, and training procedures as the pretrained model.The comparison isolates exposure to the 39 prior sessions.Source: F PRETRAINING · p. 17
  • Pretraining improved movement-related behavior decoding on every tested session by 0.046 to 0.094 R2 (mean 0.076), with choice decoding ahead on four of five sessions.The comparison uses the same target-session data budget; the models differ in exposure to 39 prior sessions.Source: F PRETRAINING · p. 17, Table F.1 · p. 17

G PHONEME AND SEMANTIC DECODING

  • Phoneme decoding maps backbone outputs to phoneme logits and uses CTC loss, which requires no time-bin-level alignment between neural activity and phoneme labels.Performance is evaluated with phoneme error rate after standard CTC decoding.Source: G PHONEME AND SEMANTIC DECODING · p. 17
  • Semantic decoding averages backbone outputs across time, maps the result to sentence-embedding space, and ranks candidate sentences by cosine similarity.The decoder uses a cosine-similarity objective, and performance is reported as top-5 sentence retrieval accuracy.Source: G PHONEME AND SEMANTIC DECODING · pp. 17–18

H ABLATION STUDY OF NEURON IDENTITY EMBEDDINGS

An ablation on held-out IBL sessions tests calibration-based neuron-identity embeddings. Including these embeddings improves decoding across the tested sessions and targets.

H ABLATION STUDY OF NEURON IDENTITY EMBEDDINGS

I ABLATION STUDY OF CHANNEL DRIFT ADAPTATION

An ablation tests calibration-based channel drift adaptation on held-out T12 days. The module yields a slight improvement in phoneme decoding and a clear improvement in sentence retrieval.

I ABLATION STUDY OF CHANNEL DRIFT ADAPTATION

  • Channel drift adaptation yielded a slight improvement in phoneme decoding and a clear improvement in sentence retrieval.The ablation compares models with and without per-day FiLM on five held-out T12 days; lower PER and higher sentence retrieval accuracy indicate better performance.Source: I ABLATION STUDY OF CHANNEL DRIFT… · p. 18, Table I.1 · p. 18

J TWO-STAGE MODEL TRAINING

This appendix specifies the shared two-stage evaluation protocol, model-capacity controls, model-specific implementations, and training procedures used for chronic-data comparisons.

J TWO-STAGE MODEL TRAINING

  • All baselines use the same pretraining corpus, then receive supervised fine-tuning for a target day or participant; SSL pretraining uses no labels, unlike supervised pretraining.The protocol is used for generalization-across-time evaluations on IBL and human chronic datasets.Source: J TWO-STAGE MODEL TRAINING · p. 18, Table J.1 · p. 19

K.1 MODEL CAPACITY CONTROL VIA SHARED BACKBONE

  • Transformer baselines share backbone depth, width, and attention-head count, while non-transformer baselines are matched to the transformer backbone's parameter count.Backbone size excludes model-specific encoders, decoder heads, and other model-specific components.Source: §K.1 · p. 19, Table K.1 · p. 19

K.2 MODEL-SPECIFIC DETAILS

  • NeuroLens infers neuron embeddings from calibration windows, averages predictor latents for trial-level targets, and uses task-specific heads and losses for choice, movement, sentence embeddings, and phonemes.IBL uses 64 calibration windows; human sessions use fewer when they contain fewer than 64.Source: §K.2 · pp. 19–20
  • CEBRA learns temporal-proximity embeddings with an InfoNCE objective and no behavioral labels, whereas Causal NDT predicts next-step neural activity without behavioral labels during pretraining.CEBRA uses a convolutional encoder; Causal NDT uses a causal Transformer and per-session read-in and output layers for IBL pretraining.Source: §K.2 · pp. 20, 22
  • Unseen-day adaptation freezes CEBRA's pretrained backbone, POYO's read-in, backbone, and read-out, and POSSM's recurrent and cross-attention modules while updating day-specific components.The updated components include a new read-in or embedding rows and decoder heads, depending on the baseline.Source: §K.2 · pp. 21, 23
  • POYO, POSSM, and SPINT are supervised decoding baselines with distinct spike-event or spike-count read-ins and shared or recurrent temporal processing.POYO tokenizes individual spikes; POSSM uses spike events with a GRU; SPINT uses per-neuron spike-count tokens and calibration-based identities.Source: §K.2 · pp. 20, 21, 23

L TRAINING DETAILS

  • Training details report a shared setup and budget across baselines, with checkpoint selection based on validation metrics to support fair comparisons.Model-specific optimizer and run details are listed separately.Source: L TRAINING DETAILS · p. 24

Related papers

Loading 2610.02864v1…