PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain-Language-Action (BLA) Models: Language-Conditioned EEG for Robotics Control

Alexandr Plashchinsky

arXiv:2608.28967v1cs.ROcs.LG

TL;DR

Direct EEG-to-action classification is limited by noisy neural signals and small fixed label spaces. Brain-Language-Action models condition neural-state interpretation on language and generate structured drone actions; the proof-of-concept reaches 0.904 average per-token evaluation accuracy across four subject-specific models.

  • Problem

    Noisy, subject-dependent EEG and small fixed label spaces constrain direct neural control, while broader command sets require more distinguishable states or another mapping mechanism.

    Source: §1 · p. 2

  • Method

    BLA models combine EEG representations with language-defined control mappings so language conditions how learned neural states are interpreted to generate robotic actions.

    Source: §1 · pp. 3–4

  • Results

    0.904 average per-token evaluation accuracy across subject-specific BLA models for subjects 1, 3, 7, and 8.

    Source: §5.3 · p. 12

  • Takeaways & Limitations

    The proof of concept supports reusing a limited set of distinguishable neural states across a broader robotic action space through language-conditioned mappings.

    Source: §7 · p. 14

    The experiments use a single motor-imagery EEG dataset and random seed, and do not demonstrate cross-subject generalization.

    Source: §6 · p. 13

Abstract

from arXiv · show

Electroencephalography (EEG)-based robotic control is commonly formulated as a direct classification problem, in which electrical neural signals are mapped to a fixed set of discrete actions. However, the limited separability and high noise of EEG signals make it difficult to scale this approach to fine-grained robotic control spaces. We introduce Brain-Language-Action (BLA) models, a framework in which language conditions the interpretation of neural representations for robotic action generation. In a BLA, a small set of reliably distinguishable brain states can be dynamically associated with different actions through a language-defined control mapping, allowing a small number of neural classes to apply to a larger global action space. We develop a proof-of-concept BLA for drone control using motor-imagery EEG from the BCI Competition IV 2a dataset. The system is trained in two stages. First, we evaluate multiple candidate EEG encoder architectures using subject-specific four-class motor-imagery classification, converting 250Hz, 3.5-second, 22-channel EEG samples into five 128-dimensional brain-token embeddings. Second, these embeddings are projected into the embedding space of a pretrained large language model (LLM) and jointly fine-tuned with language instructions to autoregressively generate structured three-token drone actions. Across 840 possible language-defined mappings between four neural states and seven flight action combinations, the resulting BLA achieves 90% per-token accuracy during evaluation. These results provide an initial demonstration that language conditioning can expand the effective control range of EEG-based robotic interfaces without requiring a corresponding increase in the number of directly distinguishable neural states.

1 Introduction

Conventional EEG control assigns neural classes to fixed commands, limiting control as robotic action spaces grow. Brain-Language-Action models use language to condition neural-state interpretation, reusing a small set of states across different actions.

1 Introduction

  • Noisy, subject-dependent EEG and small fixed label spaces constrain direct control; broader command sets require more distinguishable states or another way to specify their meanings.BCI Competition IV 2a, for example, has four motor-imagery classes.Source: §1 · p. 2
  • BLA models use language to specify the current semantics of learned neural states instead of permanently pairing each EEG class with one robot command.Given an EEG representation and language instruction, the model generates an action sequence.Source: §1 · p. 3
  • The same neural state can indicate different actions under different language-defined mappings, allowing states to be reused across a larger global action space without increasing the number simultaneously distinguishable.For example, left-hand imagery may indicate forward motion in one mapping and ascent in another.Source: §1 · p. 3
  • The proof-of-concept drone controller assigns four motor-imagery states across seven validated action sequences through 840 language-defined mappings.The system compares subject-specific four-class EEG encoders and uses the strongest encoder with a pretrained LLM to generate three structured drone-action tokens.Source: §1 · pp. 3–4

2 Brain-Language-Action Models

A BLA processes EEG representations and language instructions jointly to generate action tokens. Unlike direct EEG control, language conditions the mapping from neural class to action, allowing a small set of states to serve a larger action space.

2 Brain-Language-Action Models

  • A BLA encodes EEG, projects neural embeddings to the language model’s embedding dimensionality, and processes them with instruction tokens to generate an autoregressive action-token sequence.The action tokens represent robot commands.Source: §2 · p. 4
  • Unlike a direct EEG classifier’s fixed class-to-action mapping, a BLA conditions the mapping on language so the same neural state can represent different actions.This reuses a small cluster of distinguishable states across a larger global action space.Source: §2 · p. 4
  • The implementation generates a fixed number of discrete action tokens, one per controllable degree of freedom, for the next control step rather than a trajectory.It does not require previous robot actions or current robot state as input.Source: §2 · p. 5
  • Training first pretrains the brain encoder for EEG classification, then co-fine-tunes it with the brain-to-language projection and pretrained language model for autoregressive action generation.The classifier head is removed after pretraining.Source: §2 · p. 5

3 Brain Encoder Pretraining

The brain-encoder stage evaluates candidate architectures on subject-specific motor-imagery classification, using a common embedding interface. Adapted EEG-Inception performs strongest among the candidates and is selected for downstream BLA experiments.

3 Brain Encoder Pretraining

  • The pretraining stage identifies an encoder that converts raw multichannel motor-imagery EEG into compact, discriminative embeddings for later BLA integration.Candidate encoders are evaluated with EEG classification before being used for language-conditioned control.Source: §3 · p. 5

3.1 Dataset and Windowing

  • The study uses BCI Competition IV 2a recordings from nine subjects performing four motor-imagery tasks, sampled from 22 electrodes at 250 Hz; training is subject-specific on non-overlapping 3.5-second windows.Each window spans 875 timesteps.Source: §3.1 · p. 5

3.2 Candidate Brain Encoders

  • Nine candidate encoders span MLP, recurrent, convolutional, Transformer, subtractive, and Inception-based designs; subtractive variants learn to suppress noisy components by subtracting paired feature-extractor outputs.The Inception encoder uses parallel multi-scale convolutional processing.Source: §3.2 · p. 6

3.3 Common Embedding Representation

  • Every encoder receives the same patched input format and produces five 128-dimensional embeddings, providing a common representation interface for downstream evaluation.Each patch contains all 22 EEG channels and 175 timesteps.Source: §3.3 · p. 6

3.4 Classification Pretraining

  • A shared lightweight classifier maps each encoder’s 5 × 128 representation to four class logits, evaluating whether its embeddings contain enough discriminative information.Subject-specific models train for 30 epochs with learning rate 10^-3 and batch size 16; 10% of each subject’s windowed samples are held out for evaluation.Source: §3.4 · p. 7
  • Adapted EEG-Inception achieved the strongest classification performance among candidate architectures and was selected for downstream BLA experiments.The classifier head was discarded after pretraining, retaining the EEG encoder.Source: §3.4 · p. 7

4 BLA Architecture and Fine-Tuning

The BLA combines subject-specific EEG representations with language instructions to generate structured drone commands. Its control mappings reuse four motor-imagery states across seven action sequences, with subject-specific fine-tuning and simulation validation.

4 BLA Architecture and Fine-Tuning

  • Four subject-specific BLA models combine five 128-dimensional EEG embeddings with Qwen3-0.6B through a learned projection module.The selected EEG-Inception encoder supplies the embeddings; the projected embeddings are prepended to instruction-token embeddings, and the components are co-fine-tuned for action generation.Source: §4 · p. 7, §4.1 · pp. 7–8

4.1 Brain-to-Language Projection

  • The learned projection transforms each 128-dimensional brain embedding into Qwen3-0.6B’s 1024-dimensional token-embedding space.The projection module is learned during BLA fine-tuning rather than pretrained independently.Source: §4.1 · pp. 7–8

4.2 Drone Action Representation

  • Three autoregressive tokens encode one subsequent control step: forward/backward motion, yaw, and ascent/descent, using negative, neutral, and positive values.Each token position draws from the same three-token set; the output is a single control step, not a multi-step trajectory.Source: §4.2 · p. 8

4.3 Language-Conditioned Control Mappings

  • Language instructions assign four distinct EEG classes to four distinct actions from a vocabulary of seven, yielding 840 mappings.The EEG state and instruction must be used jointly: the signal identifies the state, while language specifies its robotic meaning.Source: §4.3 · pp. 9–10

4.4 Subject-Specific Fine-Tuning

  • Four BLA models use subjects 1, 3, 7, and 8, with each model trained and evaluated only on its own subject’s EEG.The brain encoder, projection MLP, and Qwen3-0.6B are jointly optimized for 30 epochs; 10% of subject-specific data is held out for evaluation.Source: §4.4 · p. 10

4.5 Action Sequence Simulation

  • Webots simulation validates that all 840 neural-state-to-action mappings produce valid drone commands, without using simulation in training or evaluation.The simulations use the Mavic 2 Pro drone template.Source: §4.5 · p. 10

5 Results

The selected EEGInceptionEncoder supplied subject-specific representations for BLA fine-tuning, and the resulting models achieved high per-token evaluation accuracy. Encoder performance varied across architectures, while four selected subject variants were used for downstream experiments.

5.1 Brain Encoder Pretraining Results

  • 0.627 average classification accuracy made EEGInceptionEncoder the strongest candidate, ahead of EEGSubtractiveConv2D (0.593) and EEGConv2D (0.523).Figure 1 compares average motor-imagery classification accuracy across candidate encoders and subject-specific variants.Source: Figure 1 · p. 11, §5.1 · p. 11

5.2 Best Subject Variants of the Selected Brain Encoder

  • The four selected EEGInceptionEncoder subject variants—subjects 1, 3, 7, and 8—averaged 0.786 classification accuracy and initialized the downstream BLA models.Their individual classification accuracies were 0.768, 0.873, 0.709, and 0.796, respectively; Figure 2 shows the selected subject-specific results.Source: §5.2 · p. 12, Figure 2 · p. 12

5.3 Brain-Language-Action Fine-Tuning Results

  • The results provide initial empirical support that language-conditioned mappings can expand EEG control range without increasing the number of distinguishable neural states.The authors report structured robotic action-token generation with high accuracy under these mappings.Source: §5.3 · p. 12
  • 0.904 average per-token evaluation accuracy was achieved across the four subject-specific BLA models.Their individual accuracies were 0.899, 0.899, 0.913, and 0.905; Figure 3 reports the per-token evaluation results.Source: §5.3 · p. 12, Figure 3 · p. 13

6 Discussion and Limitations

The experiments provide initial evidence that BLA can combine EEG representations with language-defined mappings to generate robotic action tokens, while remaining a proof of concept rather than general-purpose neural robotic control.

6 Discussion and Limitations

  • The subject-specific experiments support BLA as a viable approach for language-conditioned neural robotic control, but only as a proof of concept.The authors report accurate generation of structured robotic action tokens from EEG representations and language-defined mappings.Source: §6 · p. 13
  • Experiments used one motor-imagery EEG dataset and random seed, one selected brain encoder and language model, and three-token drone actions with three discrete bins per token.The architecture was closer to RT-2's discrete action-token formulation than to continuous flow-matching VLA designs.Source: §6 · p. 13
  • The study calls for evaluation across more EEG datasets, subjects, robotic embodiments, language models, brain encoders, and continuous-action generation architectures.These are identified as future-work directions.Source: §6 · p. 14

7 Conclusion

The conclusion presents BLA as a language-conditioned approach to interpreting learned EEG representations for robotic control. A subject-specific drone proof of concept supports feasibility, while cross-subject generalization and broader action spaces remain future work.

7 Conclusion

  • BLA is a robotic-control paradigm in which language conditions the interpretation of learned EEG representations.Source: §7 · p. 14
  • The subject-specific drone proof of concept projects compact brain embeddings into a pretrained language-model space and uses textual mappings to generate structured action tokens with high evaluation accuracy.Source: §7 · p. 14
  • The results support expanding EEG encoders' effective control range through language without increasing the number of directly distinguishable neural states.Source: §7 · p. 14
  • Cross-subject generalization, broader robotic action spaces, and more foundational BLA architectures remain future-work areas.Source: §7 · p. 14

A Experimental Configuration

The appendix identifies Table 2 as the configuration for brain-encoder pretraining and BLA fine-tuning, and reports the selected models, compute setup, and code availability.

A Experimental Configuration

Loading 2608.28967v1…