PaperlayerView exact revision on arXiv

Source-linked AI summary

What the LLM Should Not Say: Boundary-Aware Context Grounding for A Seven-Channel EEG Agent

Zhiyuan Xu, Yueqing Dai, Junling Li, Junwen Luo

arXiv:2606.26519v2cs.AI

TL;DR

Low-channel EEG agents need to distinguish hardware, implementation, result, and scientific boundaries. NeuraDock Agent uses deterministic local computation and versioned context grounding, improving boundary recognition and safe responses across its benchmark.

  • Problem

    EEG-specific knowledge does not ensure that an LLM understands a particular device, implementation, result schema, or scientifically justified inference.

  • Method

    NeuraDock Agent separates deterministic local EEG workflows from an LLM constrained by a versioned, hardware- and implementation-aware context pack.

  • Results

    Exact recognition of the system boundary improved from 58.3% to 79.2%, while strict safe responses increased from 26.4% to 66.7% across 36 cases, four conditions, and two models.

  • Takeaways & Limitations

    Hardware- and implementation-aware grounding calibrates which EEG requests an agent accepts, qualifies, or refuses within the tested scope.

  • Takeaways & Limitations

    The benchmark used labels authored within the current project, two provider model aliases, one generation per case, and English-language evaluation.

Abstract

from arXiv · show

Large language models (LLMs) can make scientific software easier to use. However, a general model does not automatically know which measurements a particular sensor can support, which algorithms are implemented in the current software, or which conclusions are justified by a computed result. These distinctions are especially important for low-channel electroencephalography (EEG), where sparse spatial coverage and variable signal quality make plausible but unsupported interpretations easy to produce. We present NeuraDock Agent, an open-source architecture that separates a deterministic local EEG engine from a hardware-aware language layer. The numerical engine parses recordings, performs quality control, executes reviewed spectral workflows, and writes machine-readable artifacts. The LLM receives only a compact, allowlisted summary and a versioned context pack. The context describes the seven-channel hardware, reviewed workflows, result fields, implementation boundaries, scientific limits, and reference cases. Raw EEG and dense per-sample arrays remain local We evaluate the system at three levels. First, 12 recordings produced identical structured results over ten numerical repetitions, and a complete Rest/Task run produced identical result, report, and figure hashes over three repetitions. Second, request-capture and failure-injection experiments confirmed the tested data boundary and preservation of local artifacts under HTTP, malformed-output, and connection failures. Third, a boundary-awareness benchmark tested 36 ordinary and adversarial questions under four context ablations and two LLMs, yielding 288 outputs.These results support hardware- and implementation-aware grounding as a practical mechanism for calibrating what an EEG agent accepts, qualifies, or refuses; they do not establish clinical validity or a validated absolute cognitive-load index.

1 Introduction · 2 Related Work

NeuraDock Agent addresses boundary-awareness failures in low-channel EEG by separating deterministic, versioned local computation from a hardware- and implementation-aware LLM interpretation layer. The paper frames this as a systems and evaluation contribution, not as clinical validation, on-device inference, or broad reproducibility evidence.

  • 1 Introduction: LLM access to EEG software can produce unsupported recommendations when general EEG knowledge is mistaken for knowledge of a device, protocol, implementation, or result schema.Examples include recommending frontal alpha asymmetry for a posterior-only montage, describing unavailable workflows, or turning quality flags into neurological conclusions.
  • 1 Introduction: Boundary awareness distinguishes what the montage observes, what the software implements, what output fields report, and what scientific inference those observations justify.These physical, implementation, result, and scientific boundaries are related but not interchangeable.
  • 1 Introduction: NeuraDock Agent keeps versioned local Python workflows as numerical truth while constraining the LLM with a context pack for a seven-channel, 250-Hz dry-electrode platform.The context binds the language layer to hardware, workflow registry, result meanings, implementation modules, and scientific limits.
  • 1 Introduction: The contribution is a systems and evaluation framework rather than a claim of cognitive diagnosis, expert-analysis replacement, or a clinically validated cognitive-load index.The evaluated software snapshot and model runs were completed on June 12, 2026; benchmark claims apply to that snapshot unless labeled otherwise.
  • 2.1 Deterministic EEG Software: NeuraDock Agent is a narrower workflow layer than MNE-Python and EEGLAB, explicitly constraining its hardware assumptions, inputs, outputs, and scientific claims.Its Visual Cognitive Load workflow treats alpha suppression as a relative within-recording feature, not a universal or diagnostic measure.
  • 2.2 LLM Grounding and Tool Use: Unlike grounding approaches focused mainly on retrieved documents or tool selection, the versioned hardware context layer evaluates whether the model recognizes the current capability boundary.The benchmark measures exact four-way decisions and feasible/infeasible classification, addressing over-refusal as well as hallucination.
  • 2.3 Clinical Decision Support and Medical AI: Related medical-AI and edge-AI work reinforces that broad knowledge accuracy is insufficient and that software behavior should follow the physical device contract.The evaluated numerical workflows run on a workstation, so the paper does not claim on-device LLM or TinyML inference.
  • 2.5 BCI Privacy and Neural Data: Because EEG can expose private or identity-related information, the architecture motivates data minimization, payload inspection, and auditable machine-readable outputs.NeuraDock reports include readable reports, deterministic figures, context hashes, and audit metadata; reproducibility is limited to identical outputs observed in the tested environment.

3 System Architecture

NeuraDock Agent separates local deterministic EEG computation from a constrained language layer that receives only workflow-specific summaries and versioned context. The architecture encodes seven-channel hardware, reviewed workflows, implementation boundaries, and scientific limits while preserving local artifacts when language operations fail.

  • Deterministic computation and language boundary: The local deterministic core parses recordings and writes results, reports, figures, and retained clean data, while the LLM receives an allowlisted summary and cannot modify results or execute Python.The summary is workflow-specific rather than a serialization of the complete result object.
  • Hardware and scientific boundaries: The public profile defines seven dry-electrode channels at 250 Hz covering posterior sites, with no frontal or temporal electrodes.The channels are CP5, CP6, PO3, PO4, O1, Oz, and O2; CP5 and CP6 are EEG channels, not reference channels.
  • Reviewed workflows and context: The runtime registry is closed: selecting a new numerical method requires reviewed Python code, tests, a result schema, and documentation.The context materials include the hardware profile, reviewed workflow descriptions, allowlisted fields, implementation map, scientific boundaries, and reference cases.
  • Deterministic computation and language boundary: Raw EEG samples, per-trial signals, dense window and PSD arrays, and the local source path are excluded from interpretation requests.This exposure-reduction mechanism does not establish legal de-identification or HIPAA/GDPR compliance.
  • Failure isolation: A failed LLM call does not invalidate the local workflow because the numerical report and result remain the authoritative artifacts.Planning and interpretation are optional language operations, separating computational correctness from language-service availability.

4 Visual Cognitive Load as a Constrained Case Study

The Visual Cognitive Load workflow is a relative within-recording research estimate built from posterior EEG features and within-recording normalization. Its labels and explanations are deliberately constrained: they do not establish validated cognitive states, cross-session comparisons, diagnoses, or broader psychological effects.

  • 4 Visual Cognitive Load as a Constrained Case Study: 80% of samples must pass quality control for a four-second window with a one-second step to be valid for Welch PSD analysis.The workflow extracts log10 posterior alpha power over 8–13 Hz, posterior alpha peak frequency, and normalized right-minus-left posterior alpha asymmetry from visual channels.
  • 4 Visual Cognitive Load as a Constrained Case Study: Each feature is robustly standardized within valid windows, clipped to [−3, 3], and combined into a composite score.The standardized features are log-alpha, peak frequency, and asymmetry magnitude.
  • 4 Visual Cognitive Load as a Constrained Case Study: Within-recording percentile ranks are split at score tertiles into low, medium, and high labels, which must not be treated as externally validated cognitive states.Nearly equal class counts can result mathematically from this labeling method.
  • 4 Visual Cognitive Load as a Constrained Case Study: The workflow supports relative descriptions of posterior-alpha changes but does not establish auditory load, executive control, emotion, attention, fatigue, causation, or population effects.A posterior asymmetry feature is not frontal alpha asymmetry, and percentiles are not comparable across participants or sessions without validated calibration.
  • 4 Visual Cognitive Load as a Constrained Case Study: Low-quality data requires prominent warnings, retention and excluded-window counts, and separation of observed measurements from uncertain interpretation.Blank plot regions indicate excluded windows, not zero load or a corrupted figure.

5 Evaluation

The evaluation tested software repeatability, request boundaries, failure handling, signal-quality workflows, and LLM boundary awareness. It also included preliminary human review and descriptive Rest/Task analyses, with explicit limits on reproducibility, benchmarking, and classification claims.

  • Repeatability: 12 recordings were analyzed 10 times each, while a paired Rest/Task workflow ran 3 times end to end; serialized result and artifact hashes were compared.The repeatability claim applies only to the tested software and environment, not cross-platform reproducibility.
  • Boundary and failure tests: A mock endpoint captured requests for a 7-channel recording containing 649,040 parsed EEG values, while injected HTTP 400, malformed-output, and connection-refusal failures tested local-artifact preservation.The audit assessed source paths, dense result keys, sampled raw-recording tokens, compact-summary list length, and preservation of result, report, trace, and interpretation-status files.
  • Signal-quality perturbations: 20 replicates were generated at each perturbation level for four PO3 signal-quality conditions, measuring detection, target rejection, false-positive rate, and retained-sample fraction.The fixed one-second target segment was embedded in synthetic 20-second, seven-channel signals.
  • LLM boundary benchmark: 36 cases tested six boundary categories across four context conditions, using two model aliases and an exact four-way decision-accuracy endpoint.Secondary endpoints included constraint-source F1, required-fact recall, unsupported acceptance, feasible-request rejection, and a strict safe-response indicator.
  • Evaluation limitations: The benchmark was system-derived rather than externally independent, and the four-case blinded review was preliminary, underpowered, and affected by a strong ceiling effect.A separate packet contained 96 randomized outputs for later independent review, but those ratings were incomplete and excluded from reported benchmark results.
  • Descriptive analyses: The public recording analysis used median alpha power only for descriptive visualization because external condition labels were unavailable, so it could not estimate classification accuracy.The Rest/Task pilot used six paired sessions from three participants, with participant-session pairs as the statistical unit and Task-minus-Rest posterior log10 alpha power as the prespecified contrast.

6 Results

The evaluation found repeatable numerical outputs, a tested boundary that kept raw EEG local, and preserved artifacts during language-service failures. Context ablations improved decision accuracy and calibration, while residual errors, artifact-detection gaps, and pilot analyses limited claims about autonomous or clinical use.

  • System evaluation: All 12 recordings produced one unique structured-payload hash across ten repetitions, while three end-to-end runs matched result, report, and figure hashes.These repeatability results were obtained in the tested environment.
  • System evaluation: The 8,145,499-byte raw recording was not included in captured requests, which also omitted the source path, tested dense keys, and sampled raw-value tokens.Planner and interpretation requests occupied 10,612 and 23,295 bytes, respectively; no compact list exceeded 20 entries.
  • System evaluation: Local routing selected 11 of 12 expected intents (91.7%), the LLM planner selected all 12, and tested HTTP, malformed-output, and connection failures preserved local artifacts.Preserved artifacts included results.json, report.md, agent_trace.json, and llm_interpretation.md status artifacts.
  • Quality and perturbation tests: The workflow detected tested sinusoidal perturbations at 10 µV and above but missed flatline dropouts from 1 to 15 seconds, so complete artifact detection is unsupported.The synthetic baseline produced no false-positive target detections, and detected target segments were rejected with mean retention 0.95.
  • Boundary-awareness benchmark: Pooled exact decision accuracy rose from 58.3% for Generic to 79.2% for Full context, while required-fact recall increased from 48.7% to 80.8%.The intermediate conditions scored 70.8% for Hardware and 76.4% for Hardware + implementation.
  • Boundary-awareness benchmark: Full context reduced feasible-request rejection to 8.3% with unsupported acceptance at 1.4%, but strict safe-response reached 66.7% and workflow integration remained weakest at 33.3%.Residual errors involved conditional semantics, omitted required facts, and multi-constraint orchestration; the results argue against autonomous use.

7 Discussion

The discussion frames trustworthy EEG agents as requiring useful acceptance and refusal grounded in hardware, implementation, results, and scientific limits. It also emphasizes that context selection, workflow scope, and data-boundary claims must remain auditable and appropriately limited.

  • Correctness: Scientific-agent correctness has four dimensions: numerical fidelity, physical compatibility, implementation compatibility, and inference proportional to evidence.A statistically literate answer can still violate sensor coverage, current code, or evidential limits.
  • Trustworthy behavior: Trustworthy behavior requires useful refusal for unsupported requests and useful acceptance for feasible ones, with limitations identified as physical, unimplemented, result-specific, or scientific.The preferred response should offer an available alternative rather than relying on generic caution.
  • Taxonomy: The four-way taxonomy distinguishes unsupported, not implemented, and conditional cases, making engineering support more informative than binary answers.Conditional covers plausible work requiring caveats, additional validation, or limited interpretation.
  • Context effects: Full context achieved the highest exact accuracy, whereas Hardware + implementation produced a marginally higher strict safe-response endpoint; the experiment cannot distinguish why.Possible mechanisms include distraction, over-refusal from added restrictions, or variation from a single sampled generation.
  • Context selection: Retrieval should optimize decision-relevant context rather than simply maximize relevant passages, because warnings, examples, and competing definitions can increase over-refusal or blur labels.A lightweight, versioned, deterministic or auditable policy should select question-relevant modules and expose absent context.
  • Scope and limits: NeuraDock Agent is a narrow interface to reviewed workflows, not a replacement for general EEG toolkits or expert electrophysiology, and its data-boundary evidence supports reduction and auditability rather than regulatory compliance.Broader analyses require exporting appropriate data, documented choices, expert oversight, and production safeguards such as access control and retention policies.

8 Limitations

The study’s boundary benchmark has limited independence and breadth: its labels derive from current project materials, only two changing provider aliases were tested, and each case–condition pair received one generation. Automatic fact and false-claim rules were transparent, but the supplied passage truncates before describing their full limitation.

  • Benchmark construction: The benchmark’s gold labels were not authored by an independent external group, so wording may favor the NeuraDock context pack.The benchmark was constructed from the current NeuraDock code and documentation.
  • Model coverage: Only two provider model aliases were evaluated, and aliases may change over time rather than identify immutable checkpoints.This limits interpretation across model versions or providers.
  • Sampling: Each case–condition pair received one generation, limiting assessment of output variability across repeated generations.The study evaluated one generation per case–condition pair.
  • Evaluation rules: Automatic fact and false-claim rules were transparent, but the supplied limitation passage is truncated before stating their remaining limitation.The passage ends after “transparent and repro”.

9 Future Work … Ethics and Intended Use

The paper proposes explicit, versioned boundaries for future EEG-agent development, while limiting current use to research and prototyping rather than clinical or diagnostic applications. It reports improved boundary recognition and safer responses, but emphasizes further validation, audited artifacts, and application-specific safeguards.

  • 9 Future Work: Future work should add flatline and low-variance detection, validate controlled and naturalistic recordings, and replace static context concatenation with audited selective retrieval.The retrieval work should evaluate retrieval errors and prompt size, while the pending three-reviewer benchmark should be completed and expanded.
  • 9 Future Work: Physiological validation requires counterbalanced visual tasks, external markers, behavioral and subjective measures, adequate participants, repeated sessions, and mixed-effects models before absolute cognitive-load claims.The models should separate participant, session, and trial variation.
  • 9 Future Work: Future integrations should expose versioned machine-readable contracts; robotic or medical control additionally requires latency characterization, fail-safe states, human oversight, and application-specific validation.Release 2026.6.24 provides a local online API for quality-gated interface prototypes, not medical or robotic validation.
  • 10 Conclusion: NeuraDock Agent assigns reviewed numerical computation to a deterministic local engine and uses a versioned context layer to ground language in hardware, implementation, schemas, and scientific limits.The LLM is explicitly not the EEG signal processor or source of numerical truth.
  • 10 Conclusion: 79.2% exact boundary recognition and 66.7% strict safe responses improved from 58.3% and 26.4%, respectively, across 36 cases, four context conditions, and two models.The objective is calibrated acceptance, qualification, or refusal rather than indiscriminate rejection; the ablation also indicates that context selection matters.
  • 10 Conclusion: The evidence supports hardware- and implementation-aware grounding as a promising systems mechanism, but does not establish clinical validity or replace physiological validation.The proposed principle is that language models operate inside explicit, versioned, and testable scientific boundaries.
  • Data and Code Availability: The source release, context pack, tests, API, and developer examples are available publicly, while human EEG recordings are excluded from the minimal software release.The tutorial EEG data are maintained separately from the software repository.
  • Ethics and Intended Use: The Agent is intended for research, engineering, education, and prototyping, and its software evaluations and exploratory examples do not provide medical, psychological, attention, fatigue, emotion, or performance diagnosis.The 288-output benchmark artifacts are historical research materials that should be archived immutably for independent auditing.

A Supplementary Figures · B Reproducibility Notes · C Complete Boundary-Benchmark Inventory

The supplementary material documents exploratory analyses, reproducibility artifacts, and a complete 36-case inventory for testing hardware, implementation, result, and scientific boundaries. The inventory emphasizes conditional support, explicit limitations, and refusal of unsupported or unimplemented requests.

  • A Supplementary Figures: Exploratory physiological analyses do not establish classification accuracy, an absolute cognitive-load scale, or population generalization.A preliminary four-case, one-reviewer pilot motivated the larger boundary benchmark but did not replace it.
  • B Reproducibility Notes: 288 successful JSON responses achieved 100% parse success, with raw outputs, scores, figures, tables, hashes, seeds, and checksum manifests delivered.The API key was excluded from the reproducibility deliverable.
  • C Complete Boundary-Benchmark Inventory: Table 7 inventories 36 prespecified questions with four-way gold decisions and required evidence domains: hardware, implementation, deterministic result, and scientific boundary.Conditional decisions require retaining explicit limitations; “Not implemented” means no reviewed current code path exists.
  • C Complete Boundary-Benchmark Inventory: Quality questions distinguish supported result interpretations from unsupported attentiveness claims, including 100% retention, percentile comparisons, flatline detection, and low-neighbor-correlation warnings.The inventory also asks which hardware or acquisition checks are justified for CP5 and CP6 line-noise flags.
  • C Complete Boundary-Benchmark Inventory: Scientific cases require restrained reporting: Rest/Task Alpha differences and absent quality warnings do not justify attentiveness, diagnosis, population generalization, or emotional-valence conclusions.The inventory explicitly marks diagnosing visual fatigue as unsupported and asks for the strongest defensible statement when no warnings appear.
  • C Complete Boundary-Benchmark Inventory: Operational cases test whether the current agent supports reviewed file handoffs, paired Rest/Task comparisons, polling behavior, and deployment-linked control actions.The inventory includes direct consumption of clean_eeg_data.npz, a 1.02-second update question, and a medical-robot slowdown request.
  • C Complete Boundary-Benchmark Inventory: The inventory asks developers to add review requirements before introducing numerical EEG methods and rejects immediate execution of an invented cognitive-load formula.These cases separate scientifically plausible requests from reviewed implementation paths.

D Representative Responses Across Four Context Conditions

The appendix presents four illustrative responses showing how EEG-agent answers vary with progressively richer context. The cases cover hardware observability, implementation, result interpretation, and safety-critical orchestration without permitting code execution, recording changes, or external hardware control.

  • D Representative Responses Across Four Context Conditions: The representative cases span hardware observability, implemented file handling, result and quality interpretation, and safety-critical orchestration.Each case uses the answer field from one successful kimi-k2.6 generation.
  • D Representative Responses Across Four Context Conditions: The transcripts are illustrative examples rather than additional statistical observations, with decision progression summarized using NI, C, and S labels.NI denotes not implemented, C conditional, and S supported.
  • D Representative Responses Across Four Context Conditions: Four context conditions progress from generic context to hardware-only, hardware plus implementation, and full scientific-boundary and policy context.The full condition also includes reviewed cases.
  • D Representative Responses Across Four Context Conditions: No benchmark response could execute code, modify recordings, or control external hardware.The operational consequence after each case specifies the deterministic action a safe orchestrator should permit or block.

D.1 Case OBS03: Hardware Feasibility Is Not an Implemented Workflow · D.2 Case IMP04: Supported Non-Destructive Trial Exclusion

The cases distinguish hardware relevance from implemented capability: the seven-channel montage may support SSVEP development, but no reviewed SSVEP workflow exists. By contrast, accepted NPY trial batches support one-based trial exclusion without rewriting the source file.

  • D.1 Case OBS03: Hardware Feasibility Is Not an Implemented Workflow: D.1: The seven-channel montage and 250 Hz sampling are relevant to 12–15 Hz SSVEP development, but no reviewed SSVEP workflow is implemented.The reviewed system lacks SSVEP-specific fields, modules, validated pipelines, and supplied evidence.
  • D.1 Case OBS03: Hardware Feasibility Is Not an Implemented Workflow: D.1: The appropriate response is conditional hardware feasibility and a not-implemented workflow decision.The case does not treat hardware relevance as evidence that the requested analysis can currently be run.
  • D.1 Case OBS03: Hardware Feasibility Is Not an Implemented Workflow: D.1: Requests must not be routed to a nonexistent SSVEP classifier or answered by fabricating a command.Current PSD and quality workflows remain available for exploratory sensor-level inspection, while a dedicated classifier requires a reviewed workflow, tests, schema, and validation.
  • D.1 Case OBS03: Hardware Feasibility Is Not an Implemented Workflow: D.1: Additional context does not remove every imprecision, including irrelevant aliasing discussion and dependence of frequency resolution on analysis settings.The full-context answer still contained imprecision about aliasing above 125 Hz for a 12–15 Hz request.
  • D.2 Case IMP04: Supported Non-Destructive Trial Exclusion: D.2: Trial selection is supported for accepted NPY trial-batch inputs and does not rewrite the source file.The gold decision contrasts with generic and hardware-only responses that incorrectly claimed the capability was unavailable.
  • D.2 Case IMP04: Supported Non-Destructive Trial Exclusion: D.2: The loader converts user-facing one-based trial numbers to the internal selection while preserving the original NPY file.Generic and hardware-only answers are false negatives caused by missing implementation context.

D.3 Case QUA05: Plot Gaps Are Missing Estimates, Not Zero Load · D.4 Case ORC02: Blocking Unsafe and Nonexistent Robot Control

The QUA05 gaps represent excluded, missing estimates caused by window-level quality control, not zero cognitive load. The ORC02 request must be blocked because the agent lacks the named fields, real-time robotics interface, and validated medical-control pathway.

  • D.3 Case QUA05: Plot Gaps Are Missing Estimates, Not Zero Load: The QUA05 first-plot gaps therefore answer the stated question as missing estimates created by segment quality control, rather than measurements of zero load.Only 115 windows met the validity threshold, producing disconnected valid estimates.
  • D.3 Case QUA05: Plot Gaps Are Missing Estimates, Not Zero Load: 252 of 367 windows were excluded because clean retention was below 80%, leaving 115 valid windows and disconnected Visual Cognitive Load estimates.The workflow requires at least 80% clean samples within each window.
  • D.3 Case QUA05: Plot Gaps Are Missing Estimates, Not Zero Load: 31.3% retained refers to valid windows, whereas 60.1% is the retained-sample fraction, so the metrics use different denominators.The 31.3% figure is 115/367 valid windows; overall sample retention is 60.1%.
  • D.3 Case QUA05: Plot Gaps Are Missing Estimates, Not Zero Load: The plot should preserve blank regions and exported null periods, without connecting, interpolating, or relabeling excluded windows as zero load.Interpretation should include a prominent low-retention warning and direct users to acquisition-quality checks.
  • D.4 Case ORC02: Blocking Unsafe and Nonexistent Robot Control: Not implemented: the current agent has no path to read results.json, parse vcli or confidence, or modulate robot speed through conditional logic.The reviewed system contains no such fields, file watchers, or actuation hooks, and its workflow is offline.
  • D.4 Case ORC02: Blocking Unsafe and Nonexistent Robot Control: Deployment must be blocked with no actuator command because the output is not validated as a medical control signal.Relative low/medium/high labels must not be converted into nonexistent vcli or confidence fields.
  • D.4 Case ORC02: Blocking Unsafe and Nonexistent Robot Control: Future robotics integration would require a separately validated real-time interface, formal hazard analysis, fail-safe behavior, and domain-appropriate regulatory review.Prompt-level orchestration cannot enable the integration.
Loading 2606.26519v2…