PaperlayerView exact revision on arXiv

Source-linked AI summary

EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis

Huyu Wu, Weining Weng, Yuchen Liu, Yiqiang Chen, Yang Gu

arXiv:2609.31632v1cs.LG

TL;DR

Existing EEG-agent evaluations lack broad, systematic coverage of reasoning, tool use, and workflow construction across temporal horizons. EEGAgentBench evaluates agents across six tasks with deterministic tools, revealing that current models still struggle with long-horizon EEG analysis.

  • Problem

    Existing EEG-agent evaluations cover limited tasks and narrow temporal horizons without systematically assessing reasoning, tool use, or workflow construction.

    Source: §1 · p. 2

  • Method

    EEGAgentBench evaluates 1,072 instances across six EEG tasks using ten deterministic tools that agents select to build iterative analysis workflows.

    Source: §1 · p. 2

  • Results

    Across 29 frontier LLMs, EEGAgentBench distinguishes capabilities beyond model scale and inference cost, while revealing persistent difficulty with long-horizon analysis and evidence accumulation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EEGAgentBench provides a standardized basis for assessing reasoning, tool use, and long-horizon EEG analysis capabilities.

    Source: §5 · p. 10

    Cost estimates cover 19 models and token prices, excluding tool-execution and hardware costs.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Electroencephalography (EEG) analysis is evolving from short-segment classification toward long-horizon interpretation that demands iterative evidence accumulation, multi-step reasoning, and coordinated use of specialized signal-processing tools. Although large language models (LLMs) have recently shown promise as autonomous agents for EEG analysis, existing EEG agentic evaluations remain fragmented, covering limited tasks over narrow temporal horizons with inconsistent protocols, and providing no comprehensive assessment of agents' reasoning, tool-use, and workflow construction capabilities. To address this gap, we propose \textbf{EEGAgentBench}, a unified benchmark for systematically evaluating LLM agents on short- and long-horizon EEG analysis. EEGAgentBench spans six representative EEG applications ranging from knowledge question answering to sleep staging. It encompasses signal durations from 2 seconds to nearly 23 hours, with prediction targets ranging from class labels to event intervals and epoch-level sequences. This design supports unified evaluation across knowledge reasoning, short-horizon interpretation, long-horizon event detection, and sequential understanding. The benchmark further provides 10 deterministic EEG analysis tools that expose only task-relevant signal measurements. Agents must therefore select tools autonomously, accumulate evidence iteratively, and construct multi-step workflows. For evaluation, we benchmark 29 frontier LLMs from 15 model families. Results demonstrate that EEGAgentBench effectively distinguishes agent capabilities beyond model scale and inference cost, while revealing substantial limitations of current LLM agents in long-horizon EEG analysis, particularly in sustained evidence accumulation and multi-step reasoning.

1 Introduction

Existing EEG-agent evaluations cover limited tasks and temporal horizons and do not systematically assess reasoning, tool use, or workflow construction. EEGAgentBench addresses this gap with a unified short- and long-horizon benchmark, and evaluation reveals that agents still struggle with long-horizon analysis.

1 Introduction

  • Existing EEG-agent evaluations are fragmented, with limited tasks and narrow temporal horizons and no systematic assessment of reasoning, tool use, or workflow construction.Prior efforts include EEGAgent case studies and predefined models, BrainAgent’s 60 sleep-and-emotion instructions, and traditional benchmarks built for fixed predictive pipelines.Source: §1 · p. 2
  • EEGAgentBench evaluates 29 frontier LLMs across six EEG tasks and finds that current agents still struggle with long-horizon analysis, sustained evidence accumulation, and multi-step reasoning.The benchmark also examines tool use, computational efficiency, and long-horizon failure modes.Source: §1 · p. 2
  • EEGAgentBench spans knowledge reasoning, short-horizon interpretation, and long-horizon event and sequence analysis, with recordings from 2 seconds to nearly 23 hours.Prediction targets range from class labels to event localization and sequential labeling.Source: §1 · p. 2
  • The benchmark combines six representative EEG tasks with ten deterministic, non-parametric signal-processing tools to assess knowledge reasoning, adaptive tool use, and workflow construction.Agents select tools and iteratively accumulate evidence rather than follow predefined pipelines.Source: §1 · p. 2

2 Related Work

Prior EEG analysis methods progressed from hand-crafted features to deep learning, foundation models, and LLM agents, while EEG benchmarking and interactive-agent evaluation developed along separate tracks. The paper situates EEGAgentBench at their intersection: systematic, interactive evaluation of EEG agents across diverse tasks.

2.1 EEG Analysis Methods

  • Deep learning and EEG foundation models improved performance and cross-task generalization, but remain predictor-oriented; LLM agents add tool orchestration and hierarchical analysis.EEGAgent and BrainAgent demonstrate feasibility but lack systematic evaluation across diverse EEG tasks [9] [10].Source: §2.1 · p. 3

2.2 Benchmarking EEG Analysis

  • MOABB, EEG-Bench, and NeuralBench evaluate predictive models, while existing EEG-agent studies remain demonstrations or small instruction sets rather than a unified systematic benchmark [9, 10].These benchmarks assume predefined prediction pipelines rather than interactive LLM agents [11] [12] [13].Source: §2.2 · p. 3

2.3 Interactive Evaluation of LLM Agents

  • Interactive-agent benchmarks emphasize standardized environments, verifiable outcomes, and systematic analysis of tool use and execution efficiency.Examples include AgentBench, GAIA, τ-bench, and AstaBench [14] [15] [16] [17].Source: §2.3 · p. 3

3 EEGAgentBench

EEGAgentBench combines six EEG tasks with a shared interactive environment and deterministic tools, assessing capabilities from knowledge reasoning to extended-recording analysis. Its construction filters and balances instances from public datasets into a final evaluation set.

3 EEGAgentBench

  • EEGAgentBench combines six tasks spanning knowledge reasoning, short-horizon analysis, and long-horizon analysis with an interactive environment for autonomous reasoning and adaptive tool use.The benchmark is designed to characterize capabilities across diverse EEG scenarios rather than maximize performance on individual tasks.Source: §3 · p. 3, §3.1.1 · p. 3

3.1.1 Task Design Philosophy

  • T1 tests EEG knowledge questions; T2–T4 assess artifact, epilepsy, and dementia-cohort classification; T5 detects seizure intervals, and T6 generates sleep-stage sequences.T4 is research cohort classification, not clinical diagnosis; long-horizon tasks require sustained evidence accumulation and temporally structured outputs.Source: §3.1.1 · p. 4

3.2 Interactive Environment and EEG Tools

  • Agents receive task instructions, recording metadata, and output specifications, then use a shared interface to invoke deterministic analysis tools or submit a final prediction.The environment exposes no raw waveforms, task-specific classifiers, or label information; invalid requests return standardized errors.Source: §3.2 · p. 5
  • The ten tools provide metadata, signal-quality measures, frequency- and time-domain features, temporal-window features, transient candidates, and cross-channel analyses.The same tool suite supports T2–T6, letting agents construct task-specific workflows without predefined pipelines or interfaces.Source: §3.2 · pp. 5–6

3.3 Benchmark Construction

  • The benchmark converts questions, recordings, and annotations from six public datasets into a common format containing task descriptions, recording information, and structured output specifications.The sources include MedMCQA, EEGdenoiseNet, Bonn, OpenNeuro ds004504, CHB-MIT, and Sleep-EDFx [22].Source: §3.3 · p. 6
  • Difficulty screening reduces 1,703 candidate instances to 1,490, then deterministic class balancing yields 1,072 evaluation instances.For T6, selected recordings have continuous wake epochs trimmed at the beginning and end, along with corresponding signals, to limit dominance by long wake periods.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate 29 LLMs across six EEG tasks, examining task performance, tool use, model scale, cost, and long-horizon errors. Results show uneven capabilities across tasks and substantial challenges on long recordings.

4.1 Experimental Setup

  • Models are evaluated on all 1,072 instances; T2–T6 allow iterative use of ten shared tools, and Overall equally weights six task metrics.Each model–instance pair is run once unless otherwise specified; tool trajectories are analyzed separately from task scores.Source: §4.1 · p. 7

4.2 Main Results

  • GPT-5.6-sol ranks first overall at 0.631, but task leaders differ: Llama3.3-70B leads T1, Qwen3.5-397B-A17B T2, and Qwen3.7-Max T3.GPT-5.6-sol leads T4–T6; MiniMax-M3 is the strongest open-weight model at 0.530 despite leading no individual task.Source: §4.2 · p. 7
  • T2–T3 have wide performance distributions, while T4 and T6 are shared bottlenecks; T5’s median Event-F1 is 0.284, though its top three models exceed 0.52.Two models detect no T5 events. Figure 5 displays task-level performance distributions and leaders.Source: §4.2 · p. 7, Figure 5 · p. 8
  • Tool choices vary by task, and Top-5 versus Bottom-5 median call gaps widen to 12.5 versus 3 on T5 and 13 versus 0 on T6.T2–T3 emphasize frequency- and time-domain analysis and transient candidates; T4 emphasizes band power and asymmetry, while T5–T6 shift toward windowed features.Source: §4.2 · pp. 7–8, Figure 4 · p. 7

4.3 Model Scale and Performance

  • MiniMax-M3 scores 0.530, outperforming four models with reported parameter counts of at least 1T; within-family scores rise with scale, but cross-family variation is larger.Overall spans 0.402–0.530 within the 100B–1T range; Qwen3 scores rise from 0.233 (1.7B) to 0.332 (8B), and Qwen3.5 MoE from 0.459 (35B-A3B) to 0.501 (397B-A17B).Source: §4.3 · p. 8

4.4 Cost–Performance Trade-offs

  • On the cost–performance frontier, Overall rises from 0.402 to 0.530 as cost increases from $15.4 to $93.8; GPT-5.6-sol reaches 0.631 at $723.9.MiniMax-M3 reaches 0.530 at about 13% of GPT-5.6-sol’s cost, while higher cost does not consistently predict better performance.Source: §4.4 · p. 9, Figure 6 · p. 9

4.5 Error Analysis of Long-Horizon EEG Tasks

  • On T5, GPT-5.6-sol misses 49.3% of seizure-positive recordings, while GLM-5.1 misses 29.3% but false-alarms on 96.4% of seizure-free recordings.No current model achieves both low miss and low over-alarm rates; Figure 7(a) plots these rates against one another.Source: §4.5 · p. 9, Figure 7 · p. 10
  • From recordings under 8 hours to those at least 20 hours long, 28 of 29 models lose performance, with a median Macro-F1 drop of 0.123 (44.4% relative).MiMo-V2.5-Pro is the sole exception, improving by 0.005; Figure 7 also shows representative hypnograms and duration-bin performance.Source: §4.5 · p. 9, Figure 7 · p. 10

5 Conclusion

EEGAgentBench provides a shared interactive benchmark for short- and long-horizon EEG analysis. Evaluation of 29 frontier LLMs shows that it distinguishes capabilities beyond model scale and inference cost, while exposing limitations in long-horizon analysis.

5 Conclusion

  • EEGAgentBench standardizes diverse EEG tasks, tool interactions, and evaluation protocols in a shared interactive environment for assessing agent capabilities.Source: §5 · p. 10
  • Evaluation of 29 frontier LLMs shows that EEGAgentBench distinguishes agent capabilities beyond model scale and inference cost.Source: §5 · p. 10
  • The evaluation exposes clear limitations in current agents’ long-horizon EEG analysis capabilities.Source: §5 · p. 10

A.1 Benchmark Construction and Filtering

The benchmark is built through a shared pipeline that screens candidate instances for difficulty, balances classification tasks where applicable, and applies task-specific preparation. The final set contains 1,072 instances.

A.1 Benchmark Construction and Filtering

  • The shared construction pipeline reduces 1,703 candidates to 1,490 after difficulty screening, then to 1,072 after class balancing and task-specific preparation.Source: §A.1 · p. 13
  • Difficulty screening removes instances solved by all five panel models, excluding 213 instances; T2 and full-night T6 recordings bypass this stage.The screening panel defines an instance as overly easy at a solve rate of at least 0.9, which with five models means all five solve it.Source: §A.1 · p. 13
  • Deterministic balancing targets label distributions in T3–T5, while T1 remains unbalanced and T2 retains its constructed balanced set.T3 has 188 instances, T4 has 69, and T5 has 280; T1 keeps 38 and T2 keeps 300.Source: §A.1 · p. 13
  • T3 excludes ambiguous interictal Bonn segments, limiting its epileptic class to ictal recordings; T2 mixes clean, ocular, and muscle segments across ten SNR levels.T2 has 100 segments per class, with contaminated classes spanning −7 to +2 dB.Source: §A.1 · p. 13
  • T6 trims leading and trailing wake runs to reduce wake dominance; 155 recordings are trimmed, reducing epochs by 42.7% and wake proportion from 0.637 to 0.365.Interior wake is retained, and recordings whose interior wake prevents reaching the target may retain a small wake buffer at each end.Source: §A.1 · pp. 13–14

A.2 Evaluation Metrics

The six tasks use metrics matched to their output types: classification metrics for T1–T4, event metrics for T5, and epoch-level staging metrics for T6. The section explains how these metrics handle class imbalance, chance agreement, and structured predictions.

A.2 Evaluation Metrics

  • T1 uses accuracy; T2–T4 use accuracy and macro-F1; T5 uses Event-F1 and Dice@seizure; and T6 uses macro-F1 and Cohen’s kappa.Source: §A.2 · p. 14
  • Accuracy averages per-instance correctness; random-guess baselines are 0.25 for T1, 1/3 for T2 and T4, and 0.5 for T3.Source: §A.2 · p. 14
  • Macro-F1 weights every class equally and penalizes single-class strategies; on balanced T4, always predicting AD yields about 1/3 accuracy but only 0.167 macro-F1.Source: §A.2 · p. 14
  • For T6, Cohen’s kappa discounts chance agreement; wake-heavy labels can inflate macro-F1, while hypnograms are expanded and aligned to reference length before scoring.Source: §A.2 · p. 15

A.3 Event-F1 for T5 Seizure Detection

T5 event scoring compares predicted seizure intervals with reference events using a SzCORE-style procedure. Subject-level aggregation prevents participants with more records from receiving more weight.

A.3 Event-F1 for T5 Seizure Detection

  • SzCORE-style Event-F1 evaluates detected seizure events and false alarms by comparing reference and predicted intervals, using an independent implementation with 0.1-second boundary quantization.Source: §A.3 · p. 15
  • Scoring clips and quantizes intervals, merges events separated by gaps under 90 seconds, splits events longer than 300 seconds, and extends reference events by 30 seconds before and 60 seconds after.Source: §A.3 · p. 15
  • A predicted event overlapping an extended reference event yields a true positive; unmatched reference events are false negatives, and predictions outside all extended references are false positives.Matching is not one-to-one, and extra predictions inside a reference event’s tolerance window do not add false positives.Source: §A.3 · p. 15
  • Event-F1 is computed from event counts without true negatives; CHB-MIT scores aggregate counts by subject and average Event-F1 equally across 24 subjects.A model that never detects a reference event receives Event-F1 of 0, so almost never predicting seizures is not rewarded.Source: §A.3 · p. 16

A.4 Dice@seizure for T5 Seizure Detection

Dice@seizure measures direct temporal overlap between predicted and reference seizure intervals, complementing event detection metrics that use tolerance windows. It is averaged over seizure-positive records only.

A.4 Dice@seizure for T5 Seizure Detection

  • Dice@seizure compares the durations of the merged reference and predicted intervals with their shared duration, without event-scoring tolerance windows or event merging and splitting.Intervals that overlap or touch are merged before computing the temporal overlap.Source: §A.4 · p. 16
  • A record with no reference or predicted seizure scores 1, while a seizure-positive record with no prediction scores 0.Over-wide predictions are penalized because predicted duration increases without increasing shared duration.Source: §A.4 · p. 16
  • The metric is the unweighted mean of per-record Dice across the 140 seizure-positive T5 records, with each record contributing equally.Seizure-free records are excluded because Dice does not summarize false alarms there.Source: §A.4 · p. 16

A.5 Model Parameter Counts

The model-scale analysis reports public parameter counts and context windows for evaluated models, noting gaps in reported sizes and variation in context capacity.

A.5 Model Parameter Counts

  • Reported parameter counts span 1.7B to 1,600B (1.6T); eight proprietary models lack publicly reported counts and are placed in the unknown-scale band.Dense models use total parameters; MoE counts use total and activated parameters when activation is reported.Source: §A.5 · p. 17
  • Context windows range from 32K tokens for small Qwen3 models to 1M tokens for many recent large models, and evaluation uses each model’s own window.The paper notes this matters for T5 and T6, where recordings and accumulated tool results can require tens of thousands of tokens or more.Source: §A.5 · p. 17
  • Parameter counts and context windows are taken from public model information, and decoding follows model-specific configurations rather than one shared sampling setting.Each model–instance pair is evaluated once; reported analyses are descriptive, with no hypothesis tests.Source: §A.5 · p. 17

A.6 Token Usage of the Full Evaluation

Full-evaluation token use varies widely across models, with prompt tokens usually dominating because tool results accumulate in the context over repeated turns.

A.6 Token Usage of the Full Evaluation

  • Table 5 reports prompt, completion, and total tokens, with columns rounded independently, so displayed prompt and completion counts may not sum exactly to the total.Models are sorted by descending total token use.Source: §A.6 · p. 18
  • 598.7M tokens for LongCat-2 versus 7.4M for Qwen3-1.7B: full-evaluation token use differs by about 80 times.The full evaluation covers all 1,072 instances across six tasks.Source: §A.6 · p. 18
  • For almost every model, prompt tokens make up most of total use because tool results are fed back into prompts over multiple turns, growing accumulated context.Total token use is therefore driven mainly by accumulated context rather than final-answer length.Source: §A.6 · p. 18

A.7 Model Pricing for the Cost Estimate

These appendices specify how evaluation costs are estimated, characterize task-level performance and artifact-identification behavior, and illustrate long-horizon seizure and sleep-staging cases.

A.7 Model Pricing for the Cost Estimate

  • The full-evaluation cost estimate uses input and output token counts with public per-token prices; it excludes local EEG tool execution and hardware costs.Input tokens are charged at standard rates rather than cached-input rates.Source: §A.7 · pp. 18–19

A.8 Task-Level Performance Distributions

  • T4 and T6 remain bottlenecks, with median Macro-F1 scores of 0.248 and 0.208, while T2 and T3 show broad performance distributions.T1 is comparatively mature; T5 remains limited and heterogeneous, with median Event-F1 0.284 and best score 0.549.Source: §A.8 · p. 19

A.9 SNR Dose–Response Analysis on T2 Artifact Identification

  • Artifact-identification accuracy generally rises as SNR decreases and artifacts strengthen, but models also differ in detection, clean-signal over-calling, and artifact-type confusion.Among models passing the detection gate, clean-class accuracy ranges from 0.77 to 0.05, while ocular–muscle type-confusion rates range from near zero to 0.54.Source: §A.9 · pp. 20–21

A.10 Task Case Studies

  • GPT-5.6-sol detects the second and third seizures in the one-hour case but misses the first, scoring Event-F1 0.80 with no false alarms.The trace describes screening at coarse resolution followed by finer re-examination of candidate periods.Source: §A.10 · p. 23, Figure 9 · p. 24
  • GPT-5.6-sol stages the 12.6-hour case across 1508 epochs, achieving macro-F1 0.770 and Cohen’s kappa 0.813.The trace coarsens feature windows after tool-output limits are exceeded, then produces a run-length-encoded hypnogram.Source: §A.10 · pp. 24–25, Figure 10 · p. 25
Loading 2609.31632v1…