PaperlayerView exact revision on arXiv

Source-linked AI summary

NS-Copilot: An LLM-Driven Agent System for Autonomous Neuroscience Analysis

Wuche Liu, Yiran Qiao, Linlin Hou, Rui Yang, Shusen Pu, Song Wang, Jing Ma

arXiv:2609.01971v1cs.CL

TL;DR

Neuroscience model heterogeneity and specialized data requirements make systematic model integration and analysis difficult, while existing agents lack sufficient domain-specific coordination. NS-Copilot unifies nine pre-trained models through a multi-agent, natural-language workflow that selects, executes, optimizes, and summarizes analyses. Across three benchmarks, it outperforms every baseline on the primary metric, while incurring higher execution time and computational cost.

  • Problem

    Heterogeneous neuroscience models and modality-specific constraints hinder systematic integration, selection, and evaluation, and existing agents lack the expertise to coordinate them.

  • Method

    NS-Copilot uses Planner, Coder, Controller, and Interpreter agents to orchestrate nine pre-trained models across EEG and spike modalities from raw data and natural-language task descriptions.

  • Results

    Across three benchmarks, NS-Copilot outperforms every baseline on the primary metric.

  • Takeaways & Limitations

    NS-Copilot lowers the barrier for researchers without extensive AI expertise to use advanced neuroscience models and conduct data-driven analysis.

  • Takeaways & Limitations

    Closed-loop evaluation raises execution time and computational cost, with per-run cost roughly 1.5–2× that of the single-step Codex baseline; each scenario uses one dataset.

Abstract

from arXiv · show

AI is rapidly advancing neuroscience, yet many laboratories fail to fully unleash its potential due to significant interdisciplinary barriers. While pre-trained neural models for physiological data are progressing quickly, their heterogeneous architectures and modality-specific constraints hinder systematic integration, selection, and evaluation. Despite recent advances in large language model (LLM)-based agent systems for intelligent scientific applications, existing approaches often still lack the domain expertise required to effectively select and coordinate diverse neuroscience pre-trained models and handle unique data types in this domain. We present NS-Copilot, an LLM-driven multi-agent system for neuroscience analysis that autonomously supports end-to-end workflows for diverse professional tasks. It unifies domain-specific pre-trained models and supports key neuroscience modalities, including EEG and extracellular spike data, through a natural-language interface. Given raw data and a task description, NS-Copilot orchestrates agents with specialized roles for planning, adaptive control, code generation, and result synthesis, enabling analysis without dataset-specific heuristics. We evaluate NS-Copilot on neuroscience benchmarks spanning Alzheimer's disease, Parkinson's disease, and working memory spike decoding. Across 8 trials per task, the system consistently outperforms strong baselines on the primary metric, demonstrating the ability of NS-Copilot for effective and scalable neuroscience analysis.

1 Introduction

NS-Copilot addresses barriers to integrating heterogeneous neuroscience models by providing an autonomous multi-agent pipeline for model selection, code generation, adaptive optimization, and result synthesis. Across Alzheimer’s disease, Parkinson’s disease, and working-memory benchmarks, it outperforms every baseline on each primary metric.

  • Motivation: Heterogeneous architectures and modality-specific input requirements make neuroscience model integration, selection, and evaluation difficult.Existing LLM agents also lack the domain expertise needed to coordinate specialized models and data types.
  • Contribution: NS-Copilot requires only raw neural data and a natural-language task description to conduct end-to-end neuroscience analysis.Its unified pipeline integrates multiple pre-trained models and automates analysis without dataset-specific heuristics.
  • Contribution: The Planner analyzes dataset metadata and user intent to choose an evaluation metric and prioritize compatible models.The protocol can stop early when a candidate surpasses the performance threshold.
  • Contribution: The Controller diagnoses suboptimal execution and routes optimization between targeted code modification and full strategy replanning.This closed-loop design addresses workflow-level failures beyond syntax or runtime-error repair.
  • Validation: NS-Copilot outperforms every baseline on the primary metric across benchmarks for Alzheimer’s disease, Parkinson’s disease, and working-memory decoding.The evaluation spans three distinct neuroscience tasks and neural paradigms.

2 Related Work

Prior neuroscience agents and general-purpose automation systems address parts of the analysis workflow but do not provide unified, data-aware orchestration of heterogeneous pre-trained neuroscience models. NS-Copilot is positioned to fill this integration gap across EEG and spike analysis.

  • Neural foundation models: Neural foundation models span diverse EEG architectures and are increasingly accompanied by benchmarking efforts.Existing EEG benchmarks require manual per-model configuration and do not cover spike data.
  • Scientific agents: Existing scientific agents variously orchestrate tools, search code, prepare data, or support broad biomedical tasks, but each has important scope limitations.The cited systems do not jointly provide closed-loop code execution and multi-model foundation-model orchestration for neuroscience.
  • General agents: General AutoML systems automate selection for traditional machine-learning models, while LLM-driven expert selection relies mainly on task semantics and model descriptions.These search spaces and selection signals do not fully account for domain-specific input-data characteristics.

3 Method

NS-Copilot orchestrates neuroscience analysis through Planner, Coder, Controller, and Interpreter agents, using model metadata and execution feedback to select, run, optimize, and summarize candidate pipelines. The system evaluates models sequentially, stops when a threshold is exceeded, and otherwise performs closed-loop retries.

  • 3.1 Overview: Users provide raw data and a natural-language task description, after which four agents coordinate the analysis workflow.The Planner selects models, the Coder generates executable code, the Controller optimizes failures, and the Interpreter reports results.
  • Planner: The Planner profiles modality, channel count, sampling rate, class distribution, and user intent to determine metrics and rank compatible models.Model metadata includes modality, channel compatibility, and parameter scale.
  • Model selection: A performance threshold governs early stopping; users may specify it, or the system derives it from a competitive reference score.The threshold also quantifies whether pre-trained models provide utility relative to a tuned reference.
  • Coder: The Coder generates Python for raw-file reading, preprocessing, model inference, downstream classification, and metric evaluation in a sandbox.Preprocessing can include resampling and channel selection to satisfy model-specific requirements.
  • Model interface: Nine pre-trained models across EEG and spike modalities are exposed through a unified stateless inference API with structured tool specifications.The Coder synthesizes downstream pooling and classification around latent representations.
  • Closed-loop optimization: When initial search fails, the Controller diagnoses semantic performance problems, reprioritizes the queue, and chooses Modify or Replan retries.Modify patches local algorithmic choices, whereas Replan replaces a fundamentally flawed analysis strategy.
  • Termination and reporting: The system stops immediately after a threshold-exceeding result; otherwise it continues through queued models and up to two retry rounds.Successful runs are passed to the Interpreter for metric interpretation, findings, visualizations, and recommendations.

4 Experiments

The experiments evaluate NS-Copilot across three neuroscience benchmarks, comparing it with general, scientific, and code-generation agents while examining Controller optimization, pre-trained models, and failure modes. NS-Copilot outperforms baselines on all primary metrics, while its analysis reveals both performance gains and execution trade-offs.

  • Experimental setup: Three benchmarks cover Alzheimer’s disease EEG, Parkinson’s disease EEG, and working-memory spike decoding across binary and multi-class tasks.The datasets span two modalities, 19–64 channels, and distinct neuroscience scenarios.
  • Experimental setup: 8 independent trials per benchmark measure fully autonomous execution from raw-data ingestion through final report generation.Results are reported as mean ± standard deviation across metrics.
  • Experimental setup: The Planner selects macro F1 for AD and balanced accuracy for binary PD and WM, with accuracy, AUROC, and per-class recall also tracked.Baselines include general agents, scientific agents, and code-generation systems.
  • RQ1: Overall performance: NS-Copilot outperforms state-of-the-art baselines on the primary metric for all three datasets and on eight of nine reported predictive metrics.The exception is AUROC on WM; the system also incurs higher execution time while eliminating manual pipeline engineering.
  • RQ2: Effect of the Controller: +3.80 pp is the returned Macro F1 gain over the initial AD round, with no degradation across 48 runs, although forced exploration alone changes the mean by −0.11 pp.The Controller returns the best of three rounds for each seed and routes between modification and replanning.
  • RQ3: Effect of pre-trained models: +6.82 pp is POYO-1’s held-out Balanced Accuracy gain over handcrafted spike-count features on WM, favoring the pre-trained model on 7 of 8 seeds.The advantage varies by task, from +13.34 pp on AD to +0.68 pp on PD.
  • RQ4: Error analysis: 92.3% of 325 execution attempts succeed at the code level, with 25 failures from code-generation errors or sandbox module restrictions.Failures surface explicitly and cost candidate slots while the system continues through its prioritized queue.

5 Conclusion

NS-Copilot unifies nine pre-trained models across EEG and spike modalities with automated selection, code generation, and closed-loop optimization. Across three benchmarks, it outperforms baselines on every primary metric, while ablations attribute gains to pre-trained models and best-round selection.

  • Conclusion: NS-Copilot provides a natural-language interface to nine pre-trained models across EEG and spike modalities, with automated model selection, code generation, and closed-loop optimization.The system is designed for end-to-end neuroscience analysis.
  • Conclusion: Across three benchmarks, NS-Copilot outperforms every baseline on the primary metric.The conclusion summarizes the system’s cross-task performance advantage.
  • Conclusion: Ablations indicate that gains come from pre-trained models rather than orchestration alone, while the Controller’s value lies in retaining the best round.The optimal model varies across tasks.
  • Conclusion: NS-Copilot lowers the barrier to advanced neuroscience analysis for researchers without extensive AI expertise.The stated consequence is access to state-of-the-art models and accelerated data-driven discovery.

Limitations

NS-Copilot’s closed-loop evaluation improves candidate-model exploration and selection but increases execution time and computational cost. Each scenario is represented by one dataset, limiting within-modality variation.

  • Limitations: Closed-loop evaluation increases NS-Copilot’s execution time and computational cost while supporting reliable candidate-model exploration and selection.The paper identifies this overhead as necessary for selecting the optimal method for each task.
  • Limitations: Each of the three scenarios is represented by a single dataset, leaving greater within-modality variation for future benchmarking.The authors plan to add further datasets within each scenario.

Ethics Statement

The study uses LLMs for orchestration, code generation, iterative optimization, and development assistance, with author review of generated content. It relies on publicly available datasets and models, while emphasizing that automated pipelines should complement domain expertise, especially in clinical contexts.

  • Use of Large Language Models: LLMs orchestrate model selection, analysis-code generation, and iterative optimization, while Claude also assisted with code development and grammar checking.The authors state that LLMs did not generate original research ideas or original content, and that generated content was reviewed and verified.
  • Datasets: All datasets are publicly available under open licenses, and no private or restricted-access data was used.The listed datasets include two CC0 releases and one CC-BY 4.0 release.
  • Pre-trained Models: The nine integrated models are publicly available for research use, but their weights have varying access terms, including non-commercial and gated agreements.The framework does not redistribute weights; users download them from original sources.
  • Ethics Statement: The authors report that the work does not raise any significant ethical concerns.
  • Broader Impact: Automated analysis pipelines should complement, not replace, domain expertise, particularly when incorrect classifications could have downstream clinical consequences.The authors advise users to critically evaluate all outputs.

Reproducibility Statement

The paper supports reproducibility through released source code, containerized execution, automated model-weight setup, and explicit LLM and baseline configurations. Baseline failures are reported with trial-level details and scoring conventions.

  • Reproducibility Statement: The complete multi-agent pipeline, tool specifications, and experiment scripts are provided as supplementary material and containerized via Docker.The Docker configuration is intended to enable single-command execution.
  • Reproducibility Statement: The nine model weights are publicly accessible, with setup.sh downloading them after users accept each provider’s access terms.
  • Reproducibility Statement: GPT-5.2 powers all four NS-Copilot agents, GPT-5.2-Codex establishes the performance threshold, and Claude Sonnet 4.5 serves as a baseline.Other agent baselines use their originally published default LLM configurations.
  • Baseline Failure Modes: 7/8 AD trials crashed for AutoML-Agent on raw EEG inputs, returning 0 under the reported scoring convention.AutoML-Agent also failed to produce runnable code on 5 of 8 WM spike trials.
  • Baseline Failure Modes: AutoGen produced majority-class collapse on AD, while SciAgents reached 50.00% balanced accuracy on WM and LAMBDA produced no parseable PD result.The paper reports these failures as method-specific and scores unusable trials as zero.

B Data Leakage Audit

The leakage audit checks both accession identifiers and underlying cohorts against the nine models’ pre-training corpora. None of the three benchmarks appears in those corpora, but BrainOmni had partial exposure through a related downstream benchmark.

  • Identifier and Cohort Checks: None of ds004504, ds004584, or DANDI 000006 appears in any integrated model’s pre-training corpus.The audit matched accession identifiers and also checked alternative releases of the underlying cohorts.
  • Identifier and Cohort Checks: Accession identifiers alone are insufficient because one cohort can be released under multiple identifiers.The audit therefore checked alternative releases associated with the study cohorts.
  • Model Corpus Comparisons: REVE lists 55 OpenNeuro accessions and BrainOmni lists 24, with neither list containing the study’s benchmark accessions.The nearest listed accession is ds004582, differing from ds004584 in its final digit.
  • Model Corpus Comparisons: The EEG models otherwise use corpora such as TUEG, PhysioNet, and SEED, while POYO-1 and NDT3 use non-human-primate motor-cortex data and MtM uses mouse recordings from other brain regions.The NDT3 checkpoint examined is the 200-hour configuration with eleven monkey-reaching corpora and no rodent or human source.
  • Partial Exposure: BrainOmni had partial exposure because ds004504 appeared as a 65-subject, two-class downstream subset of the 88-subject, three-class benchmark used here.The authors therefore do not describe BrainOmni as fully naive to this benchmark, despite no pre-training overlap.

C Held-out Evaluation

The held-out evaluation isolates the contribution of a fixed pre-trained model from adaptive selection by using untouched test data and paired comparisons. Results vary across tasks, with a pronounced working-memory gain but a near-tie for Parkinson’s disease.

  • +0.68 is the Parkinson’s disease gap at eight seeds, with BrainOmni favored on five of eight seeds, producing a near-tie.An earlier three-seed run reported +5.14, illustrating the small-sample effect motivating the held-out evaluation.
  • The evaluation fixes one pre-trained model, disables queue search and retries, and compares it with an identical Vanilla pipeline on untouched test folds.Both arms use the same partition for each seed, while hyperparameters are selected only from the training portion.

D Full Experimental Results

Tables 5 and 6 provide complete per-model, per-stage results for Alzheimer’s and Parkinson’s disease, while working-memory results appear in the baseline and held-out evaluations.

  • Tables 5 and 6 report per-model, per-stage results for Alzheimer’s and Parkinson’s disease, including baselines and Controller optimization trajectories.Working-memory results are reported separately in Table 2 and the working-memory row of Table 4.

E Timeouts and the BrainOmni Re-run

The BrainOmni results are strongly shaped by the uniform execution budget: timeout-induced zeros depress the standard-budget result, while a more generous allocation eliminates those failures and aligns cross-dataset rankings. The surrounding figures document the interface workflow from model search and code generation through interpretation and task completion.

  • E Timeouts and the BrainOmni Re-run: 20.76 ± 28.79 Macro F1 is BrainOmni’s initial Alzheimer’s result under the standard budget, where timeout failures enter the average as zeros.Its 4096-dimensional embeddings make BrainOmni especially sensitive to the uniform 300-second execution limit.
  • E Timeouts and the BrainOmni Re-run: Every standard deviation falls by a factor of eight or more after BrainOmni is rerun on Alzheimer’s with a more generous allocation over the same eight seeds.The reruns take 318–680 seconds per seed, and every seed produces a score.
  • E Timeouts and the BrainOmni Re-run: With the timeout lifted, BrainOmni ranks first on Alzheimer’s as well as Parkinson’s, resolving the discrepancy between the two datasets.The reported text attributes the discrepancy to the compute budget rather than the model.
  • E Timeouts and the BrainOmni Re-run: The original standard-budget figure is retained because it represents system behavior under the uniform budget, while the rerun prevents that value from being read as intrinsic to BrainOmni.
Loading 2609.01971v1…