PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain-Conditioned Action Policies for Neural Motor Decoding

Luyao Jin, Running Zhao, Huan Zhao, Vincent C. K. Cheung, Wei-Hsin Liao

arXiv:2609.34561v1cs.LG

TL;DR

Neural motor decoding remains limited by scarce paired neural-action data, despite its potential to support control through attempted movement. BrainVLA adapts a pretrained VLA policy and aligns neural representations with language; across two datasets, it outperforms evaluated baselines in cross-session decoding and task success, with high training data efficiency.

  • Problem

    Neural motor decoding is constrained by limited paired neural-action data, despite its potential to help people with paralysis control external devices through attempted movement.

    Source: §1 · p. 1

  • Method

    BrainVLA adapts a VLA policy to neural motor tasks and aligns a neural encoder with language representations so neural activity can condition the policy.

    Source: §1 · p. 2

  • Results

    Across two datasets with different action dimensions, BrainVLA outperforms evaluated baselines in cross-session decoding R2 and task success rate while demonstrating high training data efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    BrainVLA provides an approach to using existing VLA models for neural motor decoding, with strong decoding accuracy and task success on held-out sessions.

    Source: §1 · p. 2, §5 · p. 9

    BrainVLA still requires validation in real-time closed-loop BCI settings, where sensory feedback may alter neural activity.

    Source: §5 · p. 10

Abstract

from arXiv · show

Motor brain-computer interfaces (BCIs) aim to decode motor intention, enabling people with paralysis to control external devices. Neural motor decoding typically learns task-specific mappings from neural activity to kinematics, yet remains constrained by scarce paired neural-action data. We propose BrainVLA, a framework that enables neural motor decoding by drawing on a pretrained vision-language-action (VLA) model through language-mediated alignment. BrainVLA mitigates reliance on scarce paired neural-action data by leveraging VLA policies. We first construct VLA-compatible datasets including paired neural activity, action signals, language instructions, and rendered visual observations. Then, we adapt the OpenVLA-OFT policy to the target action spaces through LoRA fine-tuning. To establish an effective interface through which neural activity can convey motor intention to adapted VLA policies and guide action generation, we train a neural encoder via neural-language alignment, using language representations as semantic targets to capture latent motor intent from neural activity. The resulting neural representations serve as an endogenous intention signal to guide VLA policies to generate executable actions, while visual observations provide complementary information about the evolving task state. BrainVLA is evaluated on two neural motor datasets with different action dimensionalities using causal rollout decoding. It outperforms the evaluated baselines in cross-session decoding $R^2$ and task success rate, while demonstrating high training data efficiency. These results establish a route for neural motor decoding to draw on large-scale robotic priors through brain-conditioned VLA policies.

1 INTRODUCTION

Neural motor decoding remains constrained by scarce paired neural-action data, motivating the use of large-scale action priors through VLA policies. BrainVLA connects neural activity to an adapted VLA policy through language-mediated alignment and outperforms evaluated baselines across two datasets.

1 INTRODUCTION

  • Neural motor decoding remains constrained by limited paired neural-action data, motivating the question of whether large-scale action priors can help.Prior work has developed task-specific mappings and more generalizable neural representations, but the data constraint remains.Source: §1 · p. 1
  • VLA policies offer a potential source of action priors, but using them for neural motor decoding requires an interface that conveys motor intention.BrainVLA uses language as a semantic bridge between neural activity and the VLA policy.Source: §1 · pp. 1–2
  • BrainVLA adapts a VLA policy to target motor tasks and trains a neural encoder to align neural representations with language representations of motor intent.The aligned neural representation conditions the policy, while visual observations provide complementary task-state information.Source: §1 · p. 2
  • Across two datasets, BrainVLA outperforms evaluated baselines in cross-session decoding R2 and task success rate while showing high training data efficiency.Evaluation uses causal action decoding on datasets with different action dimensions.Source: §1 · p. 2

2 RELATED WORK

Prior neural decoding work primarily improves neural-side representations, while VLA policies offer pretrained action priors and cross-modal alignment offers a way to connect neural activity with language. BrainVLA builds on these strands by adapting OpenVLA-OFT and aligning neural and language representations.

2 RELATED WORK

  • Neural decoding approaches model, scale, or stabilize neural representations, but remain constrained by limited paired neural-action data.The cited approaches include Transformer pretraining, population-aware tokenization, and methods for cross-session variability or nonstationarity.Source: §2 · p. 2
  • VLA policies condition action generation on visual observations and language, and OpenVLA-OFT provides BrainVLA's adaptation backbone.The cited policy work includes RT-2, OpenVLA, Octo, and π0; OpenVLA-OFT is described as improving and stabilizing downstream adaptation.Source: §2 · pp. 2–3
  • Neural-language alignment draws on reported correspondence between neural and language-model representations and prior cross-modal alignment methods.Examples include CEBRA, DeWave, and BIT; BrainVLA uses language representations to capture latent motor intention and guide the VLA policy.Source: §2 · p. 3

3 METHODS

BrainVLA reformulates neural decoding as conditioning an adapted VLA policy on neural representations, with visual and proprioceptive inputs supplying task state. Its pipeline constructs multimodal training data, adapts the policy, aligns neural representations with language, and evaluates causal action rollouts.

3.1 PROBLEM FORMULATION

  • BrainVLA maps a causal neural history to an action chunk by conditioning a VLA policy, rather than directly learning a task-specific neural-to-action mapping.During adaptation, language describes the intended action; at deployment, the aligned neural representation replaces language conditioning.Source: §3.1 · pp. 3–4

3.2 VLA-COMPATIBLE DATASET CONSTRUCTION

  • The VLA-compatible dataset pairs neural activity and recorded actions with MuJoCo-rendered observations and instructions describing intended motor behavior.It includes human 7-DoF robotic-arm actions and non-human-primate 2-DoF finger-control actions, with dataset-specific task-success criteria alongside R2.Source: §3.2 · p. 4

3.3 FRAMEWORK STRUCTURE

  • The task-adapted policy uses a frozen OpenVLA-OFT backbone while LoRA adapters, an action head, and a proprioception projector are optimized.The neural encoder compresses variable-length spike sequences into latent representations that condition the downstream policy.Source: §3.3 · pp. 4–5
  • Neural-language alignment trains the neural representation against pooled language embeddings for corresponding motor intent, enabling it to replace language conditioning.The symmetric multi-positive contrastive objective treats samples sharing an instruction as positives.Source: §3.3 · p. 5

3.4 TRAINING AND INFERENCE

  • Training jointly optimizes action prediction and neural-language alignment using recorded-trajectory observations, then freezes the VLA backbone while optimizing the neural encoder.At inference, language is removed and causal neural history, current observations, and proprioception condition the policy; executed action chunks update the environment and feed back new observations.Source: §3.4 · pp. 5–6, Figure 2 · p. 5

4 RESULTS

BrainVLA is evaluated using matched causal rollout protocols, with held-out-session decoding and task success assessed alongside training-data efficiency and modality analyses.

4.1 EXPERIMENT SETUP

  • BrainVLA and reproduced baselines use the same data splits and causal rollout protocol, and performance is measured by action R2 and task success rate across held-out sessions.For an 8-action prediction chunk, inference takes 0.064 s per query on D1 and 0.045 s per query on D2.Source: §4.1 · p. 6

4.2 BRAINVLA DECODING PERFORMANCE

  • BrainVLA achieves the strongest zero-shot transfer to held-out sessions among the evaluated methods, with D2 reaching its target around the third frame despite moderate R2.The text notes that timing and subsequent motion can lower R2 while preserving task success.Source: §4.2 · p. 7

4.3 CLOSED-LOOP ROLLOUT RESULT

  • The sequential rollouts qualitatively demonstrate coherent neural-conditioned behavior over multiple steps on both datasets.Each observation is rendered from the state produced by previous predictions, so the rollout reflects accumulated policy behavior rather than independent frame-wise predictions.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4 TRAINING DATA EFFICIENCY

  • BrainVLA outperforms POYO and LSTM in trajectory R2 and task success across D1 training-data regimes, retaining meaningful performance with a small data subset.BrainVLA improves steadily as training data increases, while the baselines show lower performance and greater variability, particularly in the low-data regime.Source: §4.4 · p. 7, Figure 4 · p. 8

4.5 ABLATION STUDIES

  • Component ablations show performance degradation when task-specific VLA adaptation, neural-language alignment, visual observations, or the adapted VLA policy are removed or replaced.The ablations test these components under retraining; they do not isolate pretrained initialization from architectural capacity.Source: §4.5 · pp. 8–9, Table 2 · p. 8
  • The full model performs best across modality ablations; removing proprioception degrades D1, removing vision particularly harms D2 rollouts, and removing neural activity weakens decoding.Single-modality results indicate that neural activity supplies intention-related information, while vision supplies state feedback; proprioception alone performs poorly.Source: §4.5 · p. 9, Table 3 · p. 9

4.6 DIMENSION-LEVEL ANALYSIS

  • Across seven action dimensions, vision removal most affects translation (x, y, z), whereas neural removal more strongly affects roll and finger-related dimensions.The full model has positive R2 across all seven dimensions.Source: Figure 4 · p. 8, §4.6 · p. 9

5 CONCLUSION

BrainVLA connects neural activity to an adapted VLA policy through neural-language alignment. Across two datasets, causal rollout evaluation reports held-out-session decoding and task success, while real-time closed-loop BCI validation remains necessary.

5 CONCLUSION

  • BrainVLA connects neural activity to an adapted VLA policy through neural-language alignment, using existing VLA models for neural motor decoding.The work constructs VLA-compatible datasets and establishes a neural interface to the policy.Source: §5 · p. 9
  • Across two datasets, causal rollout evaluation reports strong held-out-session decoding and task success alongside effective performance with limited training data.Component ablations support the contributions of task-specific adaptation, alignment supervision, and the adapted policy as a whole.Source: §5 · p. 9
  • BrainVLA still requires validation in real-time closed-loop BCI settings, where sensory feedback may alter neural activity.The conclusion frames the results as an initial step toward practical neural control using adapted VLA policies.Source: §5 · p. 10

A.1 TRAINING, VALIDATION AND TEST SPLITS

The appendix describes session-level data splitting, language instructions for both tasks, and the MuJoCo reconstruction and visual observations used for D1.

A.1 TRAINING, VALIDATION AND TEST SPLITS

  • Validation sessions account for approximately 20% of held-in trials, while official held-out sessions are reserved exclusively for final testing.Hyperparameter tuning and checkpoint selection use held-in data, and held-out evaluation assesses generalization to unseen sessions.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 TASK DEFINITIONS AND LANGUAGE INSTRUCTIONS

  • D1 instructions cover seven task labels and 18 strings for object manipulation, while D2 covers initialization, outward, and return movements using 17 suggested strings.D1 descriptions distinguish wrist rotation and acquisition modes; D2 specifies index and MRS finger-group motions, including same- and opposite-direction movements.Source: §A.2 · pp. 13–14, Table 5 · p. 13, Table 6 · p. 14

A.3.1 MUJOCO ENVIRONMENT DETAILS FOR D1 DATASET

  • The D1 MuJoCo reconstruction uses a zero-gravity hand workspace with translation, wrist roll, and articulated hand joints, and provides frontal RGB observations to the VLA policy.Observations are rendered at 256 × 256 pixels and 50 Hz; the policy is queried every eight simulation steps (6.25 Hz).Source: §A.3.1 · p. 14

A.3.2 MUJOCO ENVIRONMENT DETAILS FOR D2 DATASET

The D2 appendix describes a two-degree-of-freedom finger-control reconstruction, how behavioral increments drive its joints, and how visual observations are rendered.

A.3.2 MUJOCO ENVIRONMENT DETAILS FOR D2 DATASET

  • The D2 hand model has independently controlled index and middle-ring-small finger groups, each represented by a hinge joint with a range of [0, 1.35] rad.The reconstruction uses a lightweight MuJoCo hand model.Source: §A.3.2 · pp. 14–15
  • Recorded position increments are cumulatively integrated, clipped to [0, 1], and mapped linearly to the corresponding joint range during replay.The simulator timestep is 0.02 s and gravity is zero.Source: §A.3.2 · p. 15
  • Visual observations use a fixed third-person frontal MuJoCo camera, with the scene configuration summarized in Table 8.The scene also defines front, top, and closeup cameras targeting the hand.Source: Table 8 · p. 15, §A.3.2 · p. 16

A.4 TASK-SUCCESS EVALUATION CRITERIA

Task success is scored using dataset-specific replay criteria: geometric and hand-configuration conditions for D1 and endpoint error for D2. The resulting rates describe simulated replay or kinematic success, not closed-loop control with neural-decoder feedback.

D1 criteria

  • D1 task segments receive binary success labels using task-dependent geometric and hand-configuration criteria, with all conditions in a criterion row required to hold.Reach and Grasp use minimum item distance over the segment; their proximity condition need not hold at the endpoint, and unsupported task tags are excluded.Source: Table 9 · p. 16, §A.4 · p. 16

D2 criteria

  • D2 success is evaluated in normalized 2D index- and MRS-finger position space by integrating predicted per-bin displacements from the initial position with coordinates clipped to [0,1].The integration uses N−1 updates for N prediction bins, so the final predicted displacement is not integrated; success depends only on final distance.Source: §A.4 · pp. 16–17

Aggregation

  • Overall success rate is pooled across scored D1 segments or evaluated D2 trials, rather than averaging task- or session-level rates equally.These dataset-specific criteria measure simulated replay or kinematic success and do not establish closed-loop control with feedback to the neural decoder.Source: §A.4 · pp. 16–17

B IMPLEMENTATION AND TRAINING DETAILS

BrainVLA adapts an OpenVLA backbone, encodes causal neural activity, aligns neural and language representations, and supplies neural embeddings through fixed prompt slots. The policy also receives visual observations and proprioception.

B.1 VLA ADAPTATION DETAILS

  • OpenVLA adaptation trains LoRA adapters, the action head, and proprioceptive projector while freezing the remaining backbone parameters.The action head predicts chunks of eight actions in parallel using an ℓ1 regression objective; the two datasets use nearly identical settings except for action dimensionality.Source: §B.1 · p. 17

B.2 NEURAL ENCODER

  • The POYO-style neural encoder processes spikes from the causal one-second interval before each action query, representing unit identity and token type while incorporating spike timing in attention.It adds unit-start and unit-end boundary tokens to expose recorded units, including units that did not spike in the window.Source: §B.2 · pp. 17–18

B.3 NEURAL–LANGUAGE ALIGNMENT

  • Neural and language representations are mean-pooled, ℓ2-normalized, and trained with a symmetric multi-positive contrastive objective in both neural-to-language and language-to-neural directions.Samples associated with the same instruction are positive pairs.Source: §B.3 · p. 18

B.4 FIXED-SLOT NEURAL PROMPT

  • The policy receives a fixed 53-token prompt containing 38 neural embedding slots between textual prefix and suffix, alongside RGB observation and normalized proprioception.The neural embeddings replace designated language-embedding positions; only the neural bridge is optimized in this experiment, and the auxiliary instruction string is discarded before policy input.Source: §B.4 · p. 19

C.1 BASELINE METHODS

The paper compares BrainVLA with neural-decoding and session-adaptation baselines, then details reproduced evaluation settings and causal rollout procedures. Separate language-conditioned and brain-conditioned rollouts illustrate policy behavior.

C.1 BASELINE METHODS

  • The Wiener filter linearly maps binned, optionally smoothed firing rates concatenated across temporal lags to behavioral estimates using a ridge-regularized fit.At inference, it directly maps current and preceding neural observations to predictions.Source: §C.1 · p. 19
  • The LSTM baseline models nonlinear temporal relationships by updating a recurrent state from spike counts and predicting behavior from its preceding-second history.It is trained end-to-end with mean squared error on predicted and observed velocities.Source: §C.1 · pp. 19–20
  • CycleGAN maps target-session neural activity into a source-session representation for a frozen behavioral decoder, using unpaired distribution matching and cycle consistency without target behavioral labels.The target neural activity is used for alignment at inference.Source: §C.1 · p. 20
  • NoMAD aligns new-session activity to reference latent dynamics, while SPINT infers context-dependent unit embeddings from unlabeled neural observations without gradient updates.NoMAD keeps the reference dynamics and behavioral decoder fixed; SPINT uses dynamic channel dropout to support varying population compositions.Source: §C.1 · pp. 20–21
  • NDT2 reconstructs masked spike-count tokens during multi-context pretraining, whereas NDT3 autoregressively models neural and behavioral tokens and is fine-tuned for downstream prediction.For NDT3, whether behavioral inputs are observed, predicted, or unavailable changes the decoding setting.Source: §C.1 · p. 21
  • POYO compresses variable-length spike sequences into latent representations that output queries use to generate behavioral predictions across sessions.Adapting to a new recording can involve learning new unit embeddings and fine-tuning the decoder.Source: §C.1 · p. 21

C.2 REPRODUCTION AND CAUSAL ADAPTATION DETAILS

  • Baseline reproductions share training and validation sessions and held-out calibration recordings, but local NoMAD and NDT2 variants are identified as inspired or style implementations.CycleGAN and NoMAD-inspired methods use unlabeled held-out neural activity for alignment, while SPINT uses it to infer unit identities; these are transductive calibration settings.Source: §C.2 · p. 21

D.1 LANGUAGE-CONDITIONED CLOSED-LOOP ROLLOUT RESULTS

  • Language-conditioned rollouts feed the policy's action back into MuJoCo, while brain-conditioned examples combine causal neural history with the current observation to generate each executed action.The language-conditioned protocol uses the fixed instruction and evolving visual observations; the appendix shows representative D1 and D2 brain-conditioned episodes.Source: Figure 5 · p. 22, §C.2 · p. 22, §D.2 · p. 22

D.3 TASK-LEVEL ANALYSIS

Task-level analyses show that vision and neural input contribute differently across evaluated behaviors. Spatially dependent tasks are more sensitive to vision removal, while fine-grained hand-control tasks are more sensitive to neural-input removal.

D.3 TASK-LEVEL ANALYSIS

  • Removing vision most degrades translational dimensions, while removing neural activity more strongly affects roll and upper-finger, lower-finger, and thumb control.These hand-control variables are less directly observable from the scene and depend more on intended motor commands encoded in neural activity.Source: §D.3 · pp. 22–23
  • Reach and Carry are particularly sensitive to removing vision, whereas orientation-, shape-, grasp-, and release-related tasks degrade more when neural input is removed.The full model achieves consistently high success rates across evaluated tasks.Source: §D.3 · pp. 22–23, Figure 8 · p. 24
  • The findings characterize visual observations as supporting external task-state estimation and neural activity as contributing intention-related information for subtle motor control.The analyses describe vision and neural activity as complementary sources of information.Source: §D.3 · pp. 23–24

D.4 STAGED VS. JOINT TRAINING

The staged strategy first adapts the VLA policy and then trains the neural interface; on D1 it yields stronger overall trajectory decoding and task success than joint training. On D2, BrainVLA retains decoding and task-level success with small training subsets, while Figure 11 visualizes neural–language representational similarity.

D.4 STAGED VS. JOINT TRAINING

  • The staged strategy adapts the VLA policy with language conditioning before training the neural encoder to replace that conditioning.The authors suggest language may provide a more stable semantic signal during adaptation, before neural representations are mapped into the established conditioning space.Source: §D.4 · p. 25
  • On D1, staged training yields stronger overall trajectory decoding and task success than joint training, with clearer gains in spatial dimensions and several manipulation tasks.Joint training remains competitive on some dimensions and tasks but is less consistent across the full action space.Source: §D.4 · p. 25

D.5 DATA-EFFICIENT VALIDATION ON D2

  • BrainVLA outperforms POYO and LSTM in trajectory R2 and task success across D2 data regimes, retaining meaningful performance with only a small training subset.Performance improves steadily with more data, while the baselines show lower performance and greater variability, especially in the low-data regime.Source: §D.5 · p. 25, Figure 10 · p. 25

D.6 NEURAL–LANGUAGE ALIGNMENT VISUALIZATION

  • Figure 11 compares neural and language representations by cosine similarity, with rows grouped by instruction, columns for language representations, and outlined diagonal entries marking matched pairs.Representations are mean-pooled and ℓ2-normalized before similarity is computed; warmer colors indicate higher similarity.Source: §D.6 · p. 26, Figure 11 · p. 26

Related papers

Loading 2609.34561v1…