PaperlayerView exact revision on arXiv

Source-linked AI summary

EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals

Mingxu Zhang, Ying Sun, Yuhan Li, Yang Ji, Dazhong Shen, Ke Zhang, Shan Huang

arXiv:2608.24086v1cs.AIcs.CEcs.SE

TL;DR

LLMs lack systematic evaluation for analyzing raw electromagnetic measurements, where relevant variables must be discovered from schema-free I/Q captures. EMRB benchmarks this capability through executable code across progressively difficult tasks and introduces ReconPilot to structure reconnaissance, targeted analysis, and verification. Scores decline sharply from basic measurement to system design, while ReconPilot improves overall performance across most tested backbone combinations.

  • Problem

    Existing benchmarks do not systematically test whether general-purpose LLMs can discover signals, implement analysis code, and answer engineering questions from raw electromagnetic data.

  • Method

    EMRB provides raw I/Q files and engineering questions across 200 problems, five difficulty levels, and 27 question types, evaluated through executable analysis code and deterministic verification.

  • Results

    Performance declines from 84.9% on basic measurement to 21.2% on system design, while ReconPilot raises overall scores by up to 17.6 points across three backbones.

  • Takeaways & Limitations

    The benchmark identifies multi-signal analysis and system design as the practical boundary for current models, while structured reconnaissance improves results across most tested combinations.

  • Takeaways & Limitations

    EMRB uses synthetic signals under controlled conditions and therefore does not establish performance on real field measurements with hardware impairments, multipath, or unmodeled signal types.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as code agents for scientific and engineering analysis, but their ability to analyze raw physical-layer measurements remains untested. We introduce \textbf{EMRB} (\textbf{E}lectro\textbf{m}agnetic \textbf{R}easoning \textbf{B}enchmark), which evaluates whether LLMs can analyze raw I/Q data by writing and running code. EMRB contains 200 problems across five difficulty levels and 27 question types, from signal detection to OFDM design, generated from 11 signal types with verified ground truth. Unlike benchmarks built on preprocessed features or structured tables, EMRB provides only the raw capture; the quantities each question refers to must first be discovered through code. We evaluate 14 LLMs spanning proprietary, open-weight, and reasoning-oriented families. Scores range from 24.1\% to 78.9\%, with the mean dropping from 84.9\% on basic measurement to 21.2\% on system design. We also propose \textbf{ReconPilot}, a structured method that separates signal reconnaissance, targeted analysis, and self-verification. Across three backbones, ReconPilot raises the overall score by 3.8 to 17.6 points and improves 13 of 15 backbone-level combinations tested. All data and code are publicly released in \href{https://github.com/mingxuZhang2/EMRB}{\textcolor{blue}{our GitHub repository}}.

1 Introduction

EMRB addresses the lack of systematic evaluation for LLMs analyzing raw electromagnetic signals by requiring code-based discovery and engineering reasoning. It introduces a verified benchmark and ReconPilot, while results show sharp difficulty-related performance declines and gains from structured analysis.

  • Motivation: Raw I/Q analysis remains systematically unevaluated because existing benchmarks use specialized encoders, short snippets, or preprocessed features rather than requiring discovery from raw data.The benchmark targets the full workflow of identifying signals, implementing analysis code, and answering engineering questions.
  • Contributions: EMRB contains 200 problems across five difficulty levels and 27 question types, generated from 11 signal types with verified ground truth.Problems span tasks from signal detection to OFDM design and require executable Python analysis.
  • Contributions: ReconPilot separates fixed signal reconnaissance, targeted analysis, and self-verification, raising overall scores by up to 17.6 points across three backbones.The method is intended to structure the analysis chain before and during targeted reasoning.
  • Findings: Scores range from 24.1% to 78.9%, while mean performance falls from 84.9% on basic measurement to 21.2% on system design.The reported decline reflects increasing demands for connected measurements and entity-bound analysis.

2 Related Work

Prior electromagnetic benchmarks evaluate fixed outputs or rely on specialized encoders, while code and data-analysis benchmarks lack raw physical measurements and schema-free signal interpretation. EMRB combines these settings by evaluating executable analysis of raw I/Q data.

  • EM signal benchmarks: Existing electromagnetic datasets evaluate fixed outputs such as modulation labels, device IDs, or beam indices under supervised settings.They do not ask LLMs to inspect raw measurements, write analysis code, and interpret results.
  • EM signal benchmarks: MERLIN’s EM-Bench uses language-based evaluation but still relies on a specialized signal encoder, while TeleCom-Bench targets textual comprehension rather than raw signal data.These distinctions motivate code-mediated analysis directly over measurements.
  • Code-execution benchmarks: Code-generation and agent benchmarks assess synthesis or tool use without grounding tasks in raw physical measurements.Their evaluation setting differs from signal-processing analysis over complex-valued captures.
  • Data-analysis benchmarks: Data-analysis benchmarks operate on structured tables with explicit schemas, whereas raw I/Q captures require models to construct variables through signal processing.This schema-free requirement is central to EMRB’s distinction from table-based evaluation.

3 EMRB Benchmark Design

EMRB is designed as a reproducible, multi-level benchmark that exposes progressively harder reasoning demands over raw I/Q captures. Its taxonomy moves from isolated measurements to integrated system decisions across known synthetic signal scenarios.

  • Design principles: EMRB preserves an authentic workflow in which models load raw I/Q data, select signal-processing methods, execute code, and answer from numerical results.Signals are provided as files because analysis requires procedures such as FFT, PSD estimation, autocorrelation, and peak detection.
  • Difficulty taxonomy: The benchmark contains five levels and 27 non-overlapping question types, progressing from isolated measurements to integrated engineering judgment.Each level introduces new analytical requirements rather than repeating the same task structure.
  • Difficulty taxonomy: L1 measures calibrated basic quantities, while L2 requires choosing procedures such as bandwidth definitions, FFT/windowing, autocorrelation, and STFT analysis.L1 and L2 begin with a preliminary scan summary, whereas that summary is withdrawn from L4 onward.
  • Difficulty taxonomy: L3 connects measurements to communication quantities including bitrate, E_b/N_0, PAPR, ADC quantization, and digital down-conversion.It shifts the task from characterizing signal contents to interpreting communication-system implications.
  • Difficulty taxonomy: L4 requires inventorying three to five emitters before multi-signal analysis, while L5 propagates that inventory through interference assessment and system design.Errors in the sequential L5 chain invalidate the final design.
  • Data generation: The signal library covers 11 signal types, with parameters sampled by level and several emitters sharing captures at L4 and L5.Each problem is determined by an archetype-seed pair, and reference checks verify recoverability from the released waveform.

4 Evaluation Protocol

EMRB evaluates interactive code-based analysis under a shared turn budget and deterministic scoring. Models must decide what to compute, interpret intermediate outputs, and submit quantities that satisfy entity-specific and functional acceptance rules.

  • Interaction protocol: Each model receives a question and raw I/Q file path, uses a sandboxed Python tool, and may take up to 15 interaction turns before submitting a structured answer.The turn budget includes intermediate reasoning such as inspecting a PSD and selecting a sub-band.
  • Interaction protocol: The protocol tests a complete workflow: deciding what to compute, writing executable signal-processing code, interpreting outputs, and judging measurement sufficiency.This differs from evaluating isolated code executions or subtasks.
  • Scoring: Entity-bound evaluation matches each extracted value to its sub-question, physical unit, and signal identity before applying quantity-specific tolerances.Integrated L4 and L5 designs are accepted through functional rules that allow multiple valid solutions.

5 ReconPilot: Structured Signal Analysis

ReconPilot addresses two recurring raw-I/Q analysis failures—undirected scene exploration and unchecked inconsistencies—with a three-stage workflow that maps regions, guides targeted analysis, and verifies answers.

  • Motivation: Raw-I/Q analysis requires models to discover the signal scene and solve the task, creating risks of unfocused exploration and internally inconsistent answers.Examples include bitrate–modulation contradictions and silently unanswered sub-questions.
  • Three-Stage Pipeline: The architecture therefore places fixed reconnaissance before targeted analysis and self-verification after it, and can be applied to any code-capable LLM.The three stages are illustrated as a bounded region map, guided analysis, and consistency verification.
  • Three-Stage Pipeline: Stage 1 uses deterministic reconnaissance to estimate the noise floor, detect occupied spectral regions, and report each region’s frequency limits, width, integrated power, and peak level.The reconnaissance script applies Welch PSD and a threshold 6 dB above the estimated noise floor.
  • Three-Stage Pipeline: Stage 2 supplies the region map alongside the question and raw I/Q file, letting the model focus its code-execution budget on task-specific measurements and reasoning.The map localizes where to look but does not answer the question by itself.
  • Three-Stage Pipeline: Stage 3 requires answer review without new code execution, checking sub-question coverage, unit consistency, physical plausibility, and format compliance.This stage targets silent inconsistencies introduced during multi-turn analysis.

6 Experiments

The experiments show that EMRB exposes sharply separated reasoning demands: models perform relatively well on basic measurements but struggle with multi-signal analysis and system design. ReconPilot improves performance across most tested backbones, especially where reconnaissance and verification address cascading analysis errors.

  • Overall performance: 78.9% is the highest overall model score, while 24.1% is the lowest; no model reaches 80%, leaving the benchmark unsaturated.The same Python tools do not eliminate large differences in organizing raw signal analysis.
  • Question-type analysis: L5 averages below 25 points across all three question types, while Shannon capacity reaches only 43 at L4 because extracting bandwidth and noise terms is difficult.Absolute power in dBm is the weakest L1 type at 67, and L2 energy and PSD is likewise weakest at 54.
  • Headroom validity: 98.8% to 99.8% of L1–L4 points are covered by the union of models, while oracle replay scores 100.0 at every level, showing the missing points are reachable.A reference solver scores 75.7 on L5, establishing a lower bound on attainable performance from the released data.
  • Capability profiles: 84.9% on L1 declines to 21.2% on L5, supporting EMRB’s intended progression from basic measurement to integrated system design.Mean scores are 72.3% on L2, 68.7% on L3, and 53.1% on L4.
  • Capability profiles: 31.9 points separate L4 and L5 means because later tasks require connected spectrum survey, waveform recovery, coexistence analysis, and system design.The L3-to-L4 decline is 15.6 points as multi-signal dependencies are introduced.
  • Practical boundary: The strongest models can exceed 85% on early workflow stages, but the best score for turning a crowded-band survey into system design is only 39.8%.The practical boundary appears when several emitters share the spectrum and measurements must remain attached to the correct emitter.
  • ReconPilot performance: 17.6 points is ReconPilot’s largest overall gain, and the method improves 13 of 15 backbone-level combinations tested.Overall gains are 14.0 points for DeepSeek V4 Flash, 17.6 for Gemini 3.5 Flash, and 3.8 for GPT-5.5.
  • Failure propagation: 43 points of downstream work are zeroed when an L5 inventory misses two of six emitters, illustrating how reconnaissance errors propagate through prerequisite gating.At L3, an unmeasured root symbol rate propagates into three of five sub-questions.

7 Conclusion

EMRB evaluates LLM analysis of raw electromagnetic measurements through executable code, revealing substantial difficulty as measurements become mutually dependent. ReconPilot improves overall performance by up to 17.6 points, while the benchmark remains reproducible through synthetic signals and deterministic scoring.

  • Scores across 14 LLMs range from 24.1% to 78.9% and decline from 84.9% on basic measurement to 21.2% on system design.
  • ReconPilot raises the overall score by up to 17.6 points across the evaluated backbones.The method uses fixed reconnaissance to support analysis when measurements become mutually dependent.
  • EMRB uses synthetic signals and deterministic scoring, enabling a reproducible acceptance test as models improve.

8 Limitations

EMRB’s controlled synthetic design limits what its results establish about real-world electromagnetic analysis. Its benchmark scope covers selected communication and radar waveforms under AWGN, while ReconPilot may miss weak or overlapping signals and does not resolve low L5 system-design performance.

  • EMRB does not establish model performance on real field measurements with hardware impairments, multipath, or unmodeled signal types.
  • The benchmark covers RF communication waveforms and LFM radar sweeps under AWGN rather than the full breadth of electromagnetic phenomena.
  • ReconPilot’s fixed PSD-based reconnaissance may miss weak or overlapping signals, and low L5 scores show that inventory alone does not resolve system design.
  • No human expert baseline is provided; reachability is instead bounded using oracle replay, a model union, and a programmatic reference solver.

9 Ethics and Broader Impact

EMRB uses only programmatically synthesized captures and contains no real radio recordings, personal data, or material requiring ethics review. Although raw spectrum analysis has dual-use relevance, the benchmark uses textbook signal models without parameters from deployed systems.

  • All captures are programmatically synthesized, with no real radio recordings, personal data, or material requiring ethics review.
  • EMRB has dual-use relevance to electronic warfare, but contains no parameters of deployed systems and uses textbook signal models.
  • The authors judge the marginal uplift to an adversary to be negligible.

A Datasheet for EMRB

EMRB is a reproducible, programmatically generated benchmark whose raw I/Q captures test measurement, reasoning, and system-design competence under deterministic validation. Its diagnostics reveal cascading measurement failures and bounded limitations in both the benchmark and its scoring.

  • Composition and generation: The dataset uses complex64 I/Q files, JSON metadata, 11 signal types, five levels, and 27 question types spanning measurement through system design.All 200 problems pass waveform validation, while generation parameters provide analytical ground truth.
  • Composition and generation: EMRB contains 200 programmatically generated problems with deterministic signal synthesis, question construction, and ground-truth computation.Each instance is determined by an archetype–seed pair, enabling structurally matched but numerically distinct problems.
  • Validation and scoring: Deterministic verifiers match each extracted value to its sub-question, unit, and signal identity before applying quantity-specific tolerances.Integrated designs are evaluated with functional acceptance rules, allowing multiple valid solutions rather than requiring one reference string.
  • Diagnostics: Early inventory errors propagate because signal measurements are reused in interference analysis, waveform recovery, and system design.L1 measurement scores span 18.7% to 92.9%, wider than the 52.5% to 95.8% aggregate range, so aggregate L1 scores do not isolate detection ability.
  • Reconnaissance scope: The reconnaissance script deliberately reports regions rather than source counts and omits source classification, symbol-rate estimation, chirp fitting, amplitude statistics, and autocorrelation.This deterministic, model-independent bound avoids unreliable separation of overlapping emitters but limits reconnaissance coverage.
  • Scoring caveat: Rationale checks cover 11 of 100 L2 points, while assertion-only credit could raise the overall score by at most 2.2 points.Models earned 62% of rationale points on average versus 74% of remaining L2 points.
  • Diagnostics: Absolute power is the weakest L1 type at 67 points, and L2 energy and PSD is likewise the weakest type at 54.This pattern identifies calibrated power as an early and persistent weakness across levels.

J Per-Level Failure Case Analysis

The failure cases show that EMRB difficulty arises from binding measurements to signal identities and preserving dependencies across multi-step analyses. Errors can invalidate downstream scores even when intermediate computations appear plausible.

  • L1: Signal-identity binding failure: A misplaced chirp frequency caused Llama 3.3 70B to lose credit for frequency, power, SNR, and bandwidth outputs despite detecting three emitters.The reported −9.0 MHz center frequency fell outside the ±0.5 MHz full-credit tolerance.
  • L2: Procedural fluency without conceptual grounding: L2 models often computed three bandwidth measures correctly but lost rationale credit by selecting 3-dB bandwidth without explaining occupied bandwidth’s allocation relevance.The verifier awarded the comparison points only for the occupied-power definition and its rationale.
  • L3: Cascading chain errors: A QPSK-as-BPSK misidentification halved estimated bit rate, shifted required Eb/N0 by about 3 dB, flipped link margin, and produced a zero score.The propagated error exceeded the ±3 dB tolerance at every chained sub-question.
  • L4: Multi-entity binding collapse: Swapping two correctly measured frequency ranges produced a wrong overlap verdict and invalidated downstream interference calculations.The overlap verdict gated 40% of the sub-question score.
  • L5: Catalog-gated scoring cascade: Missing one of six L5 signals zeroed the corresponding catalog entry and prerequisite-gated interference or system-design sub-questions.A weak burst emitter 8 dB below the strongest signal is given as an example of a missed source.
  • L5 extraction-filter deficit: The extraction-filter sub-question received zero on 527 of 560 attempts, primarily because models omitted or merged the required target in their inventories.Of 492 wrong-target submissions, 427 omitted the true target and 390 merged two reference emitters.

K Model Response Examples

Response examples contrast code-verified, structured analysis with plausible but weak free-form reasoning. ReconPilot improves source discovery and dependency checking, although its gains vary across models and tasks.

  • L1 comparison: GPT-5.5 performs multiple rounds of calibrated signal-processing code, whereas Llama 3.3 70B gives plausible answers from a prompt hint without code-verified measurements.
  • L5 system-design response: GPT-5.5’s L5 analysis chains inventory, interference identification, and filter design, recovering all six sources and scoring 8/8 on overlap identification.Its passband was one of four designs among 560 attempts satisfying the retention constraint.
  • L5 system-design response: The same L5 response scored 70.0/100 but reported filter behavior inconsistent with measured performance, recovered few symbols, and earned no modem-upgrade credit.The measured filter retained 0.99 of the target at 1.5 dB post-filter SIR, not the reported 0.80 at 3.7 dB.
  • L5 system-design response: The L5 figure illustrates chained reasoning in which the verifier checks each stage against ground truth and physical constraints.

L Benchmark Comparison

EMRB differs from prior EM, wireless, code, and data-analysis benchmarks by requiring raw full-length I/Q analysis through self-directed code execution. Its case studies show substantial but uneven ReconPilot gains on difficult tasks.

  • Benchmark comparison: Existing EM and wireless benchmarks evaluate fixed outputs or encoded representations, while EMRB requires raw full-length I/Q recordings and free-text engineering answers through code execution.
  • ReconPilot comparisons: ReconPilot raised Gemini 3.5 Flash from 47.0/100 to 85.9/100 on the paired L3 case.The ReconPilot run used 13 code calls versus two for free-form execution.
  • ReconPilot comparisons: Gemini’s free-form analysis guessed the root symbol rate after unsuccessful measurements, causing a 2.9× error to propagate across later quantities.ReconPilot derived the quantity from its reconnaissance region map and reached full credit on Q1, Q2, and Q5.
  • ReconPilot comparisons: On the L5 example, free-form GPT-5.5 found four of six sources, missing two overlapping emitters and mislabeling the FM carrier.
  • ReconPilot comparisons: GPT-5.5 scored 23.4/100 free-form and 65.7/100 with ReconPilot on the paired L5 problem.The ReconPilot response used nine code calls compared with three in the free-form run.
  • ReconPilot comparisons: ReconPilot’s structured division of labor recovered the fifth and sixth sources through region-map auditing and dependency checking.
Loading 2608.24086v1…