PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Data Needs Semantic Tokenization: Behavioral Events as Boundaries of Session-Transferable Tokens

Sangyoon Bae, Jiook Cha

arXiv:2610.03001v1cs.LG

TL;DR

Different neuron populations make each electrophysiology session a new input for per-neuron models, limiting shared representations across sessions. TWS turns population activity between task events into tokens without neuron or session embeddings. It decodes movement on unseen IBL sessions where a pretrained per-neuron model is at chance, and transfers from mice to macaques and Utah arrays with a linear probe.

  • Problem

    Because neuron identities typically do not overlap across sessions, per-neuron tokens are session-specific, leaving a need for representations usable across changing populations.

    Source: §1 · p. 1

  • Method

    TWS segments trials at behavioral events and converts each intervening population state into tokens without neuron or session embeddings.

    Source: §1 · p. 2

  • Results

    On unseen IBL sessions, TWS decodes movement while a per-neuron foundation model pretrained on those sessions decodes at chance; frozen TWS trained on mice transfers to macaques and Utah arrays with a linear probe.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    For cross-session generalization, the tokenizer matters more than the model on top.

    Source: §1 · p. 2, §5 · p. 9

    TWS targets Type A variables carried in shared population subspaces; choice and block remain near chance across sessions as Type B variables.

    Source: §3.1 · p. 3, §4.2 · p. 6

Abstract

from arXiv · show

Extracellular electrophysiology records a different set of neurons in every session. Neural foundation models embed each neuron and each session into their tokens, so every new session is an input they have never seen, and they fail to generalize to it. A tokenizer for new sessions needs a unit that every session shares and that carries behavior. Population activity offers such a unit. It evolves on a low-dimensional manifold that persists across neuronal turnover and across animals once sessions are aligned. This manifold changes regime at task events such as stimulus onset and movement onset. Within each regime the population occupies a state, the part of the manifold it spans between two events, and each state carries its own behavioral meaning. We propose Tokenization with States (TWS), which segments each trial, one repetition of the task, at these events and converts every state into tokens of population geometry, with no neuron or session embedding. On held-out International Brain Laboratory (IBL) sessions, TWS decodes movement even from regime boundaries that carry no information about the target, while a per-neuron foundation model pretrained on those sessions decodes at chance. Frozen after training on mice alone, TWS transfers to macaques and Utah arrays with only a linear probe. On reach direction, a target that its boundaries do not define, it achieves a Matthews correlation of 0.23, where the event time alone achieves $0.01$. For cross-session generalization, the token matters more than the model on top.

1 INTRODUCTION

Different neuron populations make each electrophysiology session a new input for per-neuron models. TWS instead tokenizes population states between behavioral events, enabling shared tokens that support movement decoding on unseen sessions and transfer across datasets.

1 INTRODUCTION

  • Session-specific neuron identities make per-neuron tokens a session-specific vocabulary because neuron overlap across sessions is typically zero.Electrode shifts, impedance changes, and spike sorting differences contribute to session-to-session population changes.Source: §1 · p. 1
  • Population activity offers a candidate shared behavioral unit because its low-dimensional manifold persists across neuronal turnover and animals, and changes regime at behavioral events.Activity between two events tracks what the animal does.Source: §1 · pp. 1–2
  • TWS turns each behavioral regime into tokens of population geometry without neuron or session embeddings, creating a token space shared across sessions.It segments trials at task events and represents each state with the subspace spanned by activity across recorded neurons.Source: §1 · p. 2, Figure 1 · p. 2
  • On unseen IBL sessions, TWS decodes movement while a per-neuron foundation model pretrained on those sessions decodes at chance.The comparison holds even when TWS's regime boundaries carry no information about the target.Source: §1 · pp. 2–3
  • A frozen TWS trained on mice transfers to macaques and Utah arrays, and its linear probe outperforms every Table 4 baseline on choice and reach direction.These targets are not defined by the token boundaries; the baselines include a per-neuron foundation model trained on the target datasets.Source: §1 · p. 3

2 RELATED WORK

Existing multi-session neural models and alignment methods retain or infer neuron or session identity, or fit parameters to new sessions. TWS instead uses states between task events without inferring unit identity or fitting new-session parameters.

2 RELATED WORK

  • TWS infers no unit identity and fits no parameter to a new session, unlike methods that retain identities, recalibrate, or align sessions.The passage describes POYO, POYO+, NDT2, NEDS, CEBRA, FALCON, alignment methods, and SPINT as using one or more of these alternatives.Source: §2 · p. 3
  • POYO, POYO+, and NDT2 learn neuron, session, or context embeddings; NEDS finetunes on new sessions, while CEBRA needs a model or input layer per session.FALCON benchmarks the recalibration these approaches require.Source: §2 · p. 3
  • Alignment methods fit a transform for each new session, whereas SPINT infers unit identity from unlabeled trials in that session.TWS's tokens are instead states between task events.Source: §2 · p. 3

3 METHOD

TWS seeks a trial representation that captures task structure despite session-specific neuron populations. It segments activity at task events, summarizes each state into permutation-invariant population tokens, and processes those tokens for behavioral decoding.

3.1 PROBLEM SETUP

  • TWS targets Type A variables carried in shared population subspaces; Type B variables with session-varying decoding directions need not survive removal of neuron identity.Locomotion and arousal are examples of Type A, while choice is an example of Type B.Source: §3.1 · p. 3

3.2 TWS: TOKENIZATION WITH STATES

  • Each variable-neuron trial is split into three behavioral regimes, each represented by two tokens of dimension 256.The six tokens form the input to the backbone and behavioral decoder.Source: §3.2 · pp. 3–4
  • Soft sigmoid gates place regime boundaries at stimulus and movement event times with trainable offsets shared across trials and sessions.The offsets account for population activity settling into a new regime after the event; the transitions between event times and shifted boundaries fall outside the three states.Source: §3.2 · p. 3
  • Each state is represented by an orthonormal basis on a Grassmannian, bounding what it can carry; omitting Gram–Schmidt changes decoding by at most 0.01.Four kernel-3 convolution blocks mix the six tokens.Source: §3.2 · p. 4
  • Permuting recorded neurons with their coordinates leaves TWS tokens unchanged, but with a quarter of the units it retains 83% of movement decoding.The aggregation has no positional encoding over neurons.Source: §3.2 · p. 4

4 EXPERIMENTS

The experiments show that TWS’s cross-session results depend chiefly on the tokenizer and its task-event boundaries, while its population-state representation also transfers across datasets. They further show that behavior and session identity can be separated and that useful reaction-time decoding is possible with only five labeled sessions.

4.1 DATASETS AND SETUP

  • Evaluation spans 243 IBL sessions and Steinmetz, Gallego macaque, and NLB datasets, using frozen embeddings and leave-one-session-out probes on IBL.Cross-dataset evaluation retains IBL weights for TWS and PerceiverIO; NLB pseudo-sessions test transfer across trials only.Source: §4.1 · pp. 4–5

4.2 TWS TOKENS DECODE UNSEEN SESSIONS WHERE PER-NEURON TOKENS COLLAPSE

  • On held-out IBL sessions, TWS leads baselines on movement and reaction time even with uninformative boundaries; choice and block remain near chance.Per-neuron models can collapse to constant predictions, while TWS fits nothing per session.Source: Table 1 · p. 5, §4.2 · pp. 5–6

4.3 REGIME BOUNDARIES DECIDE WHAT THE TOKENS CAN HOLD

  • Movement decoding degrades as boundaries shift or ignore trial events; TWS nevertheless retains neural movement signal when event times alone carry none.The boundary ablation holds the architecture fixed, and training does not repair incorrect segmentation.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 THE TOKENIZER, NOT THE BACKBONE, DETERMINES CROSS-SESSION DECODING ON IBL

  • Across tested backbones, movement decoding differs little among token-mixing models, but swapping TWS for PerceiverIO halves decoding and changes which embedding variance reflects behavior.PerceiverIO embeddings devote more variance to session identity and less to movement than TWS embeddings.Source: §4.4 · p. 6, Table 3 · p. 7

4.5 TOKENS CARRY NEURAL STATE ACROSS SPECIES AND RECORDING TECHNOLOGIES

  • Frozen TWS transfers from mice to macaques and Utah arrays with a linear probe, while event time alone explains neither choice nor reach direction.Learned state summaries transfer without preserving the time course within each regime.Source: §4.5 · pp. 6–8, Table 4 · p. 7

4.6 THE ARCHITECTURE SETS THE TOKEN SPACE, BEHAVIORAL TRAINING REGROUPS IT, PRETRAINING LOWERS SESSION IDENTITY

  • Behavioral training organizes movement states more strongly than pretraining alone, while pretraining lowers session-identity variance without changing the final score; removing session means preserves movement decoding.The architecture alone already makes movement decodable across sessions.Source: §4.6 · pp. 8–9

4.7 BEHAVIOR AND SESSION IDENTITY OCCUPY DIFFERENT DIRECTIONS

  • 0.56 against 0.55: deleting session-separating directions barely changes movement decoding, while POYO and PCA decode at chance and CEBRA loses behavior as session identity falls.Session identity and behavior occupy different embedding directions.Source: §4.7 · p. 9

4.8 WITH LABELS FROM FIVE SESSIONS, TWS OUTPERFORMS BASELINES TRAINED ON ALL OF THEM

  • Five sessions let TWS from scratch decode reaction time five times better than the best baseline trained on every session, and 25% above aligned PCA.On the same five sessions, TWS also exceeds PerceiverIO fourfold.Source: §4.8 · p. 9

5 DISCUSSION

TWS uses task-defined population states to produce comparable tokens without neuron identity or session-specific fitting. Its discussion relates this design to neuronal turnover, cross-session transfer, and the behavioral information that survives changes in recorded neurons.

A NEURON IDENTITY IS SESSION IDENTITY

  • Neuron labels do not recur across IBL session pairs, and in 342 of 361 same-subject pairs probe tracks never come within 100 µm.Chronic-array datasets also lack unit-linking fields, so per-neuron vocabularies remain session-specific in the data used by these models.Source: A NEURON IDENTITY IS SESSION IDENTITY · p. 14

5 DISCUSSION

  • TWS removes neuron identity by aggregating states over an unordered neuron set, but trained embeddings still carry 0.49 of session entropy.The discussion identifies loss of within-regime time course and reliance on recorded event times as design limits.Source: §5 · p. 9, A NEURON IDENTITY IS SESSION IDENTITY · p. 14

B REGIME BOUNDARIES VS. EVENT SEGMENTATION THEORY

  • TWS uses task-defined transitions such as stimulus and movement onset, unlike Event Segmentation Theory’s observer-internal boundaries driven by prediction error.Both mark qualitative processing changes, but their origins differ: task structure versus perceptual inference.Source: B REGIME BOUNDARIES VS. EVENT… · p. 14

C EXTENDED RELATED WORK

  • Unlike data-fitted state segmentation or session-alignment methods, TWS uses task events to make each token represent the same computational regime across sessions, at the cost of requiring event times.The paper also frames these task-defined units as an analogy to subword tokens, with domain knowledge rather than statistical frequency determining boundaries.Source: C EXTENDED RELATED WORK · pp. 14–15, D ALIGNMENT-BASED BASELINES VERSUS… · p. 15

D ALIGNMENT-BASED BASELINES VERSUS TOKENIZATION

  • TWS decodes cross-session movement and reaction time without fitting new sessions, while CEBRA and POYO remain at chance; aligned PCA transfers less well and requires session-specific rotation.TWS reaches movement MCC 0.565 and RT R2=0.402, versus aligned PCA at MCC 0.349 and R2=0.226.Source: D ALIGNMENT-BASED BASELINES VERSUS… · pp. 15–16

E EMPIRICAL VALIDATION OF THE TYPE A / TYPE B DISTINCTION

  • Across IBL and Steinmetz, movement retains more signal than choice or block when recorded neurons change; Steinmetz choice retains about 1.7 times as much as IBL choice.The index measures the share of decodable signal surviving loss of neuron identity, not signal strength.Source: E EMPIRICAL VALIDATION OF THE TYPE A /… · p. 16, Table 5 · p. 16

F TRAINING DETAILS AND HYPERPARAMETERS

TWS uses self-supervised geometry and state masking before optional behavioral finetuning, with task-state geometry shaping the token space. Ablations find stable accuracy across tested dimensions, little dependence on Gram–Schmidt, and near-orthogonality between consecutive state subspaces.

F TRAINING DETAILS AND HYPERPARAMETERS

  • Training uses three self-supervised losses for epoch-mean reconstruction, manifold separation, and near-orthogonality of consecutive state subspaces, with masked-state reconstruction learning inter-state relationships.Behavioral finetuning adds classification heads per token and regression heads on the mean token.Source: F TRAINING DETAILS AND HYPERPARAMETERS · pp. 16–17

G.1 MANIFOLD DIMENSIONALITY (kdim)

  • Movement accuracy stays within 0.003 across tested kdim values with 243 decoder-training sessions; the authors use kdim=2.The reported accuracy range is 0.864–0.867.Source: §G.1 · p. 17, Table 7 · p. 18

G.2 TOKENS WITHOUT GRAM–SCHMIDT

  • Removing Gram–Schmidt changes cross-session decoding by less than 0.01 on every target, making the Grassmannian form a convenient token realization rather than a requirement.Without that step, state tokens no longer form an orthonormal basis on the Grassmannian.Source: §G.2 · p. 18, Table 8 · p. 18

G.3 PER-STATE MANIFOLD ORTHOGONALITY

  • Geometry-only pretraining yields near-orthogonal consecutive state subspaces, with principal angles of 87.0° and 87.3° in the analyzed pretrained model.An earlier run reports 88.8° for both pairs, versus 1.5° and 5.4° without pretraining.Source: §G.3 · p. 18

H BACKBONE ARCHITECTURE

  • The backbone mixes six state tokens rather than time bins; tested mixing backbones differ by at most 0.03 movement MCC, though transfer varies across datasets.The default is a four-block kernel-3 convolutional mixer with d=256.Source: H BACKBONE ARCHITECTURE · p. 18

I IMPLEMENTATION DETAILS

The implementation is invariant to neuron ordering, but not to changes in recorded units or unit count. Controls show little effect from pooling tokens under one head, while behavioral results distinguish targets whose information survives neuron changes from those that do not.

I IMPLEMENTATION DETAILS

  • TWS tokens are unchanged by neuron permutation, but different recorded neurons change the aggregated activity, and zero-filled slots make tokens depend on unit count when fewer than 128 units are recorded.The backbone receives the Gram–Schmidt basis as well as its span.Source: I IMPLEMENTATION DETAILS · p. 18

J TWO CONTROLS ON THE HEAD AND ON THE NEURONS

  • Pooling six tokens under one linear head leaves cross-session movement and reaction-time decoding essentially unchanged; 32 units retain 83% and 64 units retain 93% of 128-unit movement decoding.The reported movement MCC is 0.562 ± 0.004 with pooled tokens versus 0.565 ± 0.006 with separate heads.Source: J TWO CONTROLS ON THE HEAD AND ON THE… · p. 19, Table 9 · p. 19

L BEHAVIORAL VARIABLE CLASS DISTRIBUTIONS

  • Five of seven behavioral variables separate no method from another because they are at ceiling, base-rate dominated, or unpredictable across sessions by construction.Choice and block remain near chance across sessions, although within-session per-neuron methods confirm their information exists in raw data.Source: L BEHAVIORAL VARIABLE CLASS… · p. 19, Table 11 · p. 20

M FULL RESULTS ACROSS ALL BEHAVIORAL VARIABLES

  • Choice and block (P ≈0.52) are Type B and at chance across sessions; always predicting engaged gives 93.8% accuracy, yet no method decodes engagement across sessions (MCC ≈0); wheel velocity R2 ≈0 across methods.Source: L BEHAVIORAL VARIABLE CLASS… · p. 19

K GLOSSARY OF BEHAVIORAL VARIABLES

  • The study defines Type A targets as recoverable from any sufficiently large neuronal subset and Type B targets as dependent on which specific neurons are recorded.Feedback is structurally Type A, but its cross-session decodability is dominated by correlation with movement.Source: Table 10 · p. 20

N TRAINING ON NEW DATASETS: INITIALIZATION AND FROZEN ENCODERS

A frozen TWS trained on IBL is evaluated on new datasets with a linear probe, while related analyses compare continuous-variable decoding and state geometry. The results show mixed cross-dataset effects of IBL training, limited CEBRA regression without per-session standardization, and distinct geometry across task states.

N TRAINING ON NEW DATASETS: INITIALIZATION AND FROZEN ENCODERS

  • With both encoders trained on IBL behavior and frozen, TWS leads on choice and reach direction while PerceiverIO leads on NLB movement; IBL pretraining helps NLB but slightly lowers Steinmetz and Gallego scores.The comparison uses held-out-session MCC, and the new-dataset protocol holds out sessions by fold.Source: N TRAINING ON NEW DATASETS:… · pp. 20–22, Table 13 · p. 21

O CEBRA REGRESSION PERFORMANCE

  • Without per-session RT standardization, CEBRA reaches cross-session R2=-0.191 for RT and -0.057 for wheel velocity.Self-supervised CEBRA shows the same pattern, with R2=-0.188 and -0.056, respectively.Source: O CEBRA REGRESSION PERFORMANCE · p. 22

P PHYSIOLOGICAL VARIABLE DECODING

  • Across five held-out video sessions, TWS decodes whisker motion twice as well as PerceiverIO, but no model decodes pupil diameter across sessions.No model was trained on either physiological variable, and the authors characterize the five-session result as directional only.Source: P PHYSIOLOGICAL VARIABLE DECODING · p. 22, Table 14 · p. 22

Q STATE MANIFOLD CHARACTERISTICS

  • Consecutive state subspaces are nearly orthogonal, while the non-adjacent pair shares structure; the stimulus-to-movement state has the highest participation ratio at 2.5.Figure 4 presents session-demeaned state-centroid PCA and cumulative explained variance with participation-ratio markers.Source: Q STATE MANIFOLD CHARACTERISTICS · p. 22, Figure 4 · p. 23

R REGIME BOUNDARY INFORMATION PATHWAY ANALYSIS

The boundary pathway analysis separates timing information supplied by gate placement from trial-specific neural content, then examines how the backbone and tokenizer shape transfer and representation geometry. The results show that neural dynamics contribute movement timing, while learned embeddings organize behavioral structure and retain timing aligned to regimes.

R REGIME BOUNDARY INFORMATION PATHWAY ANALYSIS

  • Neural-only R2=0.274 exceeds gate-only R2=0.149 and is larger in 195 of 226 sessions, indicating trial-specific neural dynamics carry more movement timing than gate geometry.The decomposition is within-session on raw state means, not directly comparable to cross-session R2.Source: R REGIME BOUNDARY INFORMATION PATHWAY… · p. 24, Table 15 · p. 24
  • Event-time-only Ridge scores movement MCC 0.908 and RT R2=0.443, versus TWS-from-scratch scores of 0.565 and 0.402, respectively.Movement and RT share their defining event with the boundary, so these scores measure retained regime-aligned timing rather than neural decoding beyond the boundary.Source: R REGIME BOUNDARY INFORMATION PATHWAY… · pp. 24–25
  • After excluding the 11% of trials with movement before stimulus onset, TWS movement MCC rises from 0.565 to 0.723, near the event-time score of 0.776.Adding the boundary to representations leaves TWS movement MCC only +0.001 above boundary alone (p=0.23) and RT R2 only +0.006 above it (p=0.56).Source: R REGIME BOUNDARY INFORMATION PATHWAY… · p. 24, Table 16 · p. 24, Table 17 · p. 25

S BACKBONE SWEEP

  • On Steinmetz and NLB, the convolutional backbone scores 0.155 ± 0.006 for choice and 0.378 ± 0.008 for movement; Mamba backbones are within 0.03, while the Transformer scores 0.122 ± 0.054 and 0.299 ± 0.060.On reach direction, the Transformer scores 0.201 ± 0.006 versus 0.232 ± 0.010 for the convolutional backbone.Source: S BACKBONE SWEEP · p. 25
  • TWS embeddings separate movement states more than PerceiverIO embeddings (1.03–1.88 versus 0.48–0.72), and scratch-trained TWS backbone pairs have linear CKA 0.87–0.92 versus 0.44–0.64 for PerceiverIO.Session variance shares are 23% and 18% for TWS with convolutional and Transformer backbones, versus 31% and 28% for those backbones on PerceiverIO tokens.Source: S BACKBONE SWEEP · p. 25, Figure 5 · p. 26

T CODING RATE REDUCTION OF THE REPRESENTATIONS

  • Movement-grouped coding-rate reduction rises from 0.05 to 0.84 for TWS after scratch training, while pretraining and finetuning lowers TWS session ΔR from 12.7 to 7.0 and retains movement ΔR of 0.61.The paper cautions that absolute values depend on ϵ and overall embedding volume, so only large differences are compared.Source: T CODING RATE REDUCTION OF THE… · pp. 25–26

U CROSS-DATASET TRANSFER CONTROLS

The cross-dataset controls compare identity-free population summaries and tokenizers, and test whether pretraining requires intact temporal order. PerceiverIO transfers comparably to TWS on several tasks, while time-shuffled pretraining performs at least as well as intact pretraining in the reported evaluations.

U CROSS-DATASET TRANSFER CONTROLS

  • Adding the time course to population Level raises NLB movement by a third and doubles reach direction, raises choice tenfold, and lowers feedback.PerceiverIO retains time-course information in its patches and leads TWS on NLB movement, while TWS leads on reach direction.Source: U CROSS-DATASET TRANSFER CONTROLS · p. 27, Table 19 · p. 27
  • PerceiverIO transfers as well as TWS on the tested cross-dataset tasks: it leads on Steinmetz choice and NLB movement, TWS leads on reach direction, and they do not differ on Steinmetz feedback or movement.Untrained PerceiverIO already decodes choice at 0.169; random-bias TWS reaches three quarters of trained TWS on choice and reach direction, while training adds on choice, reach direction, and NLB movement.Source: U CROSS-DATASET TRANSFER CONTROLS · p. 28, Table 20 · p. 28
  • Time-shuffled pretraining transfers at least as well as intact pretraining, scoring slightly higher on Steinmetz choice, feedback, and reach direction, and equally on both movement targets.The shuffle preserves each trial's spike counts while removing temporal order and within-trial co-firing.Source: U CROSS-DATASET TRANSFER CONTROLS · p. 28, Table 21 · p. 29

V EVENT-TIME INFORMATION ON CROSS-DATASET TASKS

The event-time controls distinguish tasks defined by the first-movement boundary from tasks that are not. Boundary position alone decodes IBL movement and reaction time, but does not account for cross-dataset choice decoding by TWS.

V EVENT-TIME INFORMATION ON CROSS-DATASET TASKS

  • The first-movement event defines IBL movement and reaction time, whereas other-dataset movement depends on wheel or cursor speed and choice and reach direction are unrelated to boundary timing.The paper reports that the boundary alone decodes IBL movement and RT better than any model, but does not define the other datasets' targets.Source: V EVENT-TIME INFORMATION ON… · p. 29, Table 22 · p. 29
  • TWS decodes Steinmetz choice both within movement and no-movement groups, while boundary position alone decodes choice in neither group; TWS also leads on go-trial left-versus-right choice.TWS (random init) and PCA remain near chance among trials without a movement.Source: V EVENT-TIME INFORMATION ON… · p. 29, Table 24 · p. 30

W PRETRAINING LOSS ABLATION

The ablation finds little benefit from pretraining-loss differences after finetuning on 243 IBL sessions, while additional analyses assess residual movement decoding and session-level comparisons. The reported comparisons show TWS beating PerceiverIO in most held-out sessions, but neither representation adding to boundary-only movement or reaction-time decoding.

W PRETRAINING LOSS ABLATION

  • With 243 sessions, finetuned pretraining configurations converge closely: movement accuracy spans 0.016 (0.873–0.889) and feedback spans 0.008 (0.817–0.825).Training from scratch reaches 0.889 movement accuracy against 0.873 for the full pipeline; on smaller datasets, the IBL checkpoint helps on NLB but not Steinmetz or Gallego et al.Source: W PRETRAINING LOSS ABLATION · p. 30, Table 25 · p. 30

X RESIDUAL DECODING: ORTHOGONALITY OF TASK AND SESSION INFORMATION

Y SESSION-LEVEL STATISTICS

  • Across held-out sessions, TWS beats PerceiverIO in most sessions, but neither adds to boundary-only decoding on movement or reaction time.The session-level comparisons average each session's score over five model seeds and use paired differences, bootstrap intervals, and a Wilcoxon signed-rank test.Source: Y SESSION-LEVEL STATISTICS · pp. 30–31, Table 27 · p. 31
  • Across five model seeds, scratch-trained TWS + CNN has session identifiability 0.489 ± 0.017 and cross-session movement MCC 0.565 ± 0.006; at random initialization these are 0.150 ± 0.004 and 0.274 ± 0.018.The held-out sessions are fixed across the seed comparison.Source: Y SESSION-LEVEL STATISTICS · p. 31

Z SESSION DIRECTIONS AND BEHAVIOR DECODING DIRECTIONS

Across held-out IBL sessions, session information is distributed across embedding dimensions, while behavior-decoding directions carry little session identity. TWS tokens group by computational state rather than lab, and held-out trial embeddings retain cross-session behavioral structure.

Z SESSION DIRECTIONS AND BEHAVIOR DECODING DIRECTIONS

  • Session directions are principal directions of session means after regression removes embedding variation explained by movement and reaction time.Behavior-decoding directions are Ridge-probe weights for movement and reaction time, fit on the 190 training sessions.Source: Z SESSION DIRECTIONS AND BEHAVIOR… · p. 31
  • −3% movement decoding change for both TWS and PerceiverIO after removing session directions, versus 37% and 17% reaction-time losses, respectively.The two behavior-decoding directions identify sessions no better than two random directions.Source: Table 28 · p. 31, Z SESSION DIRECTIONS AND BEHAVIOR… · p. 31
  • Session information is spread across embedding dimensions: top session directions carry more information than random directions, but removing them does not lower session identification.Any two random directions identify sessions above chance; behavior-decoding directions carry no more session information than two random directions.Source: Z SESSION DIRECTIONS AND BEHAVIOR… · pp. 31–32, Table 29 · p. 32
  • TWS state-token coordinates group by state (silhouette 0.53), not lab (−0.08), subject (−0.49), or session (−0.62).The t-SNE coordinates cover state tokens from all 243 IBL sessions, including the 53 held out from training.Source: Z SESSION DIRECTIONS AND BEHAVIOR… · p. 32, Figure 8 · p. 32

TRIAL EMBEDDINGS OF HELD-OUT SESSIONS

  • For held-out trials, pretrained and fine-tuned TWS mixes sessions while separating movement behavior; cross-session movement-label sharing is +0.13, versus +0.03 for random-init TWS and +0.02 for CEBRA.POYO and PCA separate sessions; random-init TWS and CEBRA overlap across sessions only by chance.Source: Figure 9 · p. 33
  • On Steinmetz 2019, frozen IBL-trained TWS retained a third of per-neuron session information and decoded movement best, while per-neuron features decoded near chance.TWS with random initialization retained almost no session information and decoded movement nearly as well as trained TWS.Source: TRIAL EMBEDDINGS OF HELD-OUT SESSIONS · p. 33

A SHARED DECODING DIRECTION EXPLAINS DATA EFFICIENCY

Across six models, five-session decoding tracks the consistency of behavior-decoding directions, not session identification. This relationship also appears within each model, though stronger behavior signals may contribute to it.

A SHARED DECODING DIRECTION EXPLAINS DATA EFFICIENCY

  • Movement directions are computed from the difference between mean embeddings for trials with and without a first movement within 2 s; reaction-time directions use within-session covariance.Direction consistency compares each session's unit decoding direction with the mean direction of other held-out sessions.Source: A SHARED DECODING DIRECTION EXPLAINS… · pp. 33–34
  • Within each model, sessions with more consistent decoding directions also score higher on five-session decoding (ρ=0.58 to 0.80, all p<10^-5).The authors note that stronger behavior signals could contribute to this within-model relation, so they base the claim on the across-model comparison.Source: A SHARED DECODING DIRECTION EXPLAINS… · p. 34
  • Spearman ρ=0.94 for each target: across six models, five-session decoding follows direction consistency, but not session identification (ρ=−0.09 for movement; 0.03 for reaction time).The comparison uses direction consistency, five-session decoding, and session identification on 53 held-out IBL sessions.Source: Table 31 · p. 34, A SHARED DECODING DIRECTION EXPLAINS… · p. 34

Related papers

Loading 2610.03001v1…