EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals

• Unlike existing EM benchmarks [29], EMRB requires models to discover signals in raw captures, implement analysis code, and answer engineering questions rather than classify snippets or use task-specific encoders.Source: §1 · p. 1
• Raw I/Q inputs have no schema, so models must compute question-relevant entities and measurements before analysis, with downstream conclusions inheriting discovery errors.As connected measurements accumulate, the stated bottleneck is keeping each computed variable attached to the correct entity, rather than domain knowledge.Source: §1 · p. 1
• EMRB comprises 200 problems across five difficulty levels and 27 question types, spanning 11 signal types and raw-I/Q tasks from signal detection to OFDM design.Problems use verified ground truth, generated signals, deterministic scoring, and a code-execution protocol.Source: §1 · pp. 1–2
• ReconPilot separates fixed signal reconnaissance, targeted analysis, and self-verification, raising overall scores by up to 17.6 points and improving 13 of 15 backbone-level combinations.The method was tested across three backbones.Source: §1 · p. 2
• Across 14 LLMs, scores ranged from 24.1% to 78.9%, while performance fell from 84.9% on basic measurement to 21.2% on system design.Source: §1 · p. 2

EM signal benchmarks

• RadioML, WiSig [14], WASD, DeepMIMO [1], and DeepSense 6G [2] evaluate fixed outputs, while EM-Bench [29] uses a specialized encoder and TeleCom-Bench [32] targets text.EMRB instead requires workflows over raw I/Q measurements; Table 6 in Appendix L summarizes the differences.Source: §2 · p. 2

Code-execution and data analysis benchmarks

• HumanEval [6], MBPP [5], APPS [15], DS-1000 [21], AgentBench [22], ToolBench [28], GAIA [23], and MLAgentBench [17] test coding or tool use, not raw physical measurements.DA-Bench [16] and TableBench [30] use structured tables; raw I/Q time series lack schemas and require models to construct variables through signal processing.Source: §2 · p. 2

Deep learning for EM signal processing

• EM deep-learning methods [25] [26] [35] address single tasks, while RF-GPT [39], RadioLLM [7], and PReD [13] use learned encoders or multimodal models.EMRB complements these approaches by testing general-purpose LLM analysis of raw I/Q files through code execution without domain-specific fine-tuning.Source: §2 · p. 2

3.1 Design Principles

• EMRB preserves an authentic workflow, tests competence from basic measurement to system-level reasoning, and uses seeded signals with known ground truth.The stated design principles make problems reproducible and answers objectively verifiable.Source: §3.1 · p. 2

3.2 Code-Execution Evaluation Protocol

• Models analyze raw I/Q files by writing Python code, observing numerical outputs, and refining answers over multiple turns.The prompt provides the question and file path; the signal is loaded and analyzed as data rather than serialized into the prompt.Source: §3.2 · p. 2, Figure 1 · p. 3

3.3 Difficulty Taxonomy

• EMRB comprises 200 problems across five levels and 27 non-overlapping question types, with 40 problems per level.The levels progress from isolated measurements to integrated engineering judgment, rather than repeating the same task at higher parameter complexity.Source: §3.3 · p. 2
• L1–L3 progress from prior-informed measurements to method selection and communication-system interpretation.L1 uses a preliminary scan summary and typically one analysis step; L2 requires choosing procedures such as FFT/windowing or bandwidth definitions, while L3 covers bitrate, Eb/N0, PAPR, ADC quantization, and digital down-conversion. Bandwidth may mean 3-dB, null-to-null, or occupied-power width, each requiring a different implementation.Source: §3.3 · pp. 2–3
• L4 requires inventorying three to five emitters before downstream analysis, while L5 links inventory and interference assessment to system design.L4 draws five sub-questions from nine types, including burst analysis, interference assessment, and Shannon capacity; L5 covers spectrum survey, radar coexistence, and OFDM design, where errors earlier in the chain invalidate the design.Source: §3.3 · p. 3

3.4 Signal Library and Data Quality

• A parameterized library generates 11 signal types and validates answers against released waveforms and target-parameter tolerances.Seeded archetypes determine signal composition and question selection; varying seeds can produce fresh problems. Reference scripts recover scored quantities within 1% on all 200 problems, and waveforms meet power ±0.5 dB, SNR ±1 dB, and bandwidth ±5% tolerances.Source: §3.4 · p. 3

4.1 Code-Execution Loop

• Models analyze raw I/Q files through sandboxed Python for up to 15 interaction turns, then submit structured answers, testing an end-to-end workflow rather than isolated code tasks.The turn budget allows intermediate reasoning that may not involve code, such as deciding which sub-band to inspect. The model must choose computations, write and interpret signal-processing code, and judge when measurements are sufficient.Source: §4.1 · pp. 3–4

4.2 Deterministic Scoring System

• EMRB uses deterministic, entity-bound scoring against ground truth, feasibility constraints, or conclusion checks; functional rules allow valid integrated designs, while L5 gates downstream credit on prerequisites.L1–L4 each have five 20-point sub-questions, while L5 has three worth 34, 33, and 33 points. Verifier versions and task provenance fingerprints are stored with results so scores are rejected when questions, waveforms, or rules change; rationale-content credit appears only in L2 and totals 2.2% of benchmark points.Source: §4.2 · p. 4

5 ReconPilot: Structured Signal Analysis

• ReconPilot wraps LLM-guided targeted analysis with fixed reconnaissance before analysis and self-verification before submission.This structure targets broad, undirected exploration and unchecked contradictions or missing answers.Source: §5 · p. 4, Figure 2 · p. 4

5.1 Three-Stage Pipeline Design

• Stage 1 uses a shared deterministic script to map occupied spectral regions without identifying sources or resolving their contents.It estimates the noise floor with Welch PSD, detects regions at 6 dB above it, and reports frequency limits, width, integrated power, peak level, and per-region STFT statistics; region count is explicitly not source count.Source: §5.1 · p. 4
• Stage 2 gives the model the region map alongside the question and raw I/Q, preserving its unconstrained code-execution loop for task-specific analysis.The map avoids repetitive initial spectral exploration but does not determine source counts, measurement procedures, or task-specific calculations.Source: §5.1 · p. 4
• Stage 3 requires a pre-submission review without new code execution, checking answer coverage, unit consistency, physical plausibility, and format compliance.The review is intended to catch inconsistencies such as a bitrate that conflicts with the identified modulation order, or a missing sub-question answer.Source: §5.1 · p. 4

6.1 Experimental Setup

• The evaluation tests 14 LLMs across proprietary, open-weight, reasoning-oriented, and lightweight families, giving each the same problem inputs, raw-file access, Python interface, and 15-turn budget.ReconPilot is also evaluated with DeepSeek V4 Flash, Gemini 3.5 Flash, and GPT-5.5 backbones.Source: §6.1 · p. 5

6.2 Overall Performance

• Across 14 evaluated models, scores range from 24.1% to 78.9% (a 54.8-point spread): GPT-5.5 leads, while Llama 3.3 70B scores 24.1%; no model reaches 80%.GPT-5.5 reaches 78.9% with 3.7 code calls per problem, versus Grok-4.5’s 77.8% with 12.6 calls. All three L5 question types average below 25 points; calibrated power measurement is weakest at L1 and L2, and Shannon capacity is weakest at L4.Source: §6.2 · pp. 5–6

6.3 Headroom Validity

• Oracle replay scores 100.0 at all five levels, while the union of 14 models reaches 98.8%–99.8% at L1–L4 but only 53.2% at L5.A classical-DSP reference solver scores 75.7 at L5, close to twice the best model score of 39.8, establishing a lower bound on attainable performance from the released data; the solver declines to answer 18 points of sub-questions.Source: §6.3 · pp. 6–7, Table 1 · p. 7

6.4 Capability Profiles Across EMRB

• Mean scores decline across levels from 84.9% at L1 to 21.2% at L5, with a 15.6-point L3-to-L4 drop and a further 31.9-point L4-to-L5 drop.L2 and L3 impose comparable demands by cross-model mean, but individual models can score higher at L3 than L2, distinguishing formula application from selecting a method for raw data. L4 most sharply separates models (73.1-point range; σ=17.6), while even the best L5 score is 39.8%.Source: §6.4 · p. 7

6.5 ReconPilot Performance

• ReconPilot raises overall scores by 14.0 points for DeepSeek V4 Flash, 17.6 for Gemini 3.5 Flash, and 3.8 for GPT-5.5, improving 13 of 15 backbone-level combinations.The comparison holds the problem set, scoring procedure, and model configuration fixed. Gains cluster on multi-signal levels; the two exceptions are losses of 3.0 points at L1 for DeepSeek V4 Flash and 3.1 at L3 for GPT-5.5, where free-form scores already exceeded 91%.Source: Table 2 · p. 7, §6.5 · pp. 7–8

6.6 Stage Ablation

• Reconnaissance alone raises Gemini 3.5 Flash’s average from 57.2 to 68.8, verification alone to 70.2, and both stages together to 74.8.The combined gain of 17.6 is below the 24.6 points expected if the gains were independent, indicating overlapping failure repairs. At L4, reconnaissance alone scores 77.9 and verification costs 5.7 points; at L5, reconnaissance alone scores 11.0, while verification catches implausible inventories before they propagate.Source: Table 3 · p. 8, §6.6 · p. 8

6.7 Failure Case Analysis

• The five levels expose a progression from misbinding measurements to signals at L1, through weak justification at L2 and cascading errors at L3, to multi-emitter tracking failures at L4 and system-design collapse at L5.In an extraction-filter audit, 527 of 560 attempts score zero; deficits include merging overlapping emitters, omitting constraints, and failing to check whether designs meet requirements. Across levels, failures stem from losing a coherent, entity-bound chain of evidence from raw samples to answers, rather than simply lacking formulas.Source: §6.7 · p. 8

7 Conclusion

• Scores across 14 LLMs range from 24.1% to 78.9% and fall from 84.9% on basic measurement to 21.2% on system design.The decline reflects models’ difficulty maintaining an entity-bound analysis state as measurements become mutually dependent.Source: §7 · p. 8
• ReconPilot raises the overall score by up to 17.6 points, showing that fixed reconnaissance can substitute for missing capability.Source: §7 · p. 8
• Using only synthetic signals and deterministic scoring, EMRB can serve as a reproducible acceptance test as models improve.Source: §7 · p. 8

8 Limitations

• Because EMRB covers 11 synthetic signal types under AWGN, its results do not establish performance on field measurements with hardware impairments, multipath, or unmodeled signals.Its scope covers RF communication waveforms and LFM radar sweeps, not the full breadth of electromagnetic phenomena.Source: §8 · p. 8
• ReconPilot’s fixed PSD-based reconnaissance may miss weak or overlapping signals, and every backbone remains low on L5, so a stable inventory alone does not resolve system design.Source: §8 · p. 8
• EMRB has no human expert baseline; reachability is instead established by oracle replay, a model union, and a programmatic reference solver, bounding attainability from below.Source: §8 · p. 8

9 Ethics and Broader Impact

• All captures are synthesized from textbook signal models, contain no personal data or deployed-system parameters, and are judged to provide negligible adversarial uplift despite raw spectrum analysis’s dual-use relevance.The authors state that the dataset requires no ethics review.Source: §9 · p. 8

A Datasheet for EMRB

• EMRB contains 200 problems across five levels and 11 signal types, generated from seeded archetypes that determine each waveform, question, and ground-truth answer.Templates and generation scripts enable deterministic reproduction and fresh problems from unused seeds; ground truth is computed analytically or with reference signal-processing code.Source: A Datasheet for EMRB · p. 11
• All 200 problems pass recoverability validation with less than 1% numerical error, but EMRB omits real-world channel and receiver effects.The benchmark measures analysis on well-conditioned captures, not representative field environments; it is not intended for real-world signal detection.Source: A Datasheet for EMRB · pp. 11–12
• Scores span 24.1% to 78.9% without saturation, and ReconPilot improves three backbones by 3.8 to 17.6 points.The benchmark also surfaces level- and task-specific weaknesses that aggregate accuracy can hide.Source: A Datasheet for EMRB · p. 12

B.1–B.2 L1 and L2 Examples

• L1 covers signal detection and basic measurement, while L2 requires distinct signal-processing procedures, including separate implementations for three bandwidth definitions and correct emitter attribution.The L1 example uses PSD peak detection, band-power integration, and spectral and envelope cues; the L2 example includes FFT, autocorrelation, and spectrogram analysis.Source: §B.1 · p. 13, §B.2 · p. 13

B.3–B.4 L3 and L4 Examples

• L3 chains measured quantities into communication-theory calculations, while L4 requires jointly inventorying four emitters and binding each measurement to the correct signal.In L3, a wrong modulation order can propagate through bitrate, Eb/N0, and link margin; L4 combines symbol-rate, chirp, spectral-gap, and capacity questions.Source: §B.3 · p. 13, §B.4 · p. 13

B.5 L5 Example

• L5 integrates spectrum inventory, waveform recovery, and system design, with early measurement errors propagating across the problem.Its questions cover signal overlaps and channel packing, chirp characterization and recovery of an overlapped waveform, and link comparisons, power allocation, and OFDM design.Source: §B.5 · p. 14

C Complete Problem Instance

• L1–L3 prompts provide a preliminary source scan, but the L1 sample-only measurement core spans 18.7% to 92.9%, versus 52.5% to 95.8% for aggregate scores.The prompt supplies enough information for some L1 points to be answered without measurement; from L4 onward, source enumeration becomes scored and the anchor is withdrawn.Source: C Complete Problem Instance · p. 14

D.1–D.3 Prompts

• ReconPilot keeps the free-form system prompt fixed, prepends a localization-only region map, and reserves three of 15 turns for self-verification.The verification prompt checks completeness, units, physical reasonableness, and output format; the free-form setup provides Python execution and NumPy, SciPy, and Matplotlib.Source: §D.2 · p. 14, §D.1 · p. 14, §D.3 · p. 16

E ReconPilot Reconnaissance Details

• ReconPilot’s Stage 1 map reports occupied regions, not emitters, and deliberately avoids source counts, modulation labels, symbol rates, and chirp fits.It produces deterministic plain-text output shared across backbones; relative thresholds and follow-up instructions flag regions for targeted inspection without assigning source identities.Source: E ReconPilot Reconnaissance Details · p. 16

F Deterministic Scoring Criteria; F.1 Categorical Answer Matching

• Scoring uses quantity-specific units and tolerances, checks structured answers against physical constraints, and resolves categorical modulation aliases such as 4-QAM for QPSK.Waveform recovery allows documented phase, scale, and conjugation equivalences; multiple valid numerical designs can pass without an LLM judge.Source: F Deterministic Scoring Criteria · p. 16, §F.1 · p. 16

F.2 Decision and Rationale Criteria

• Four L2 rationale criteria award 11 points, or 2.2% of the benchmark, for correctly directed asserted claims, but checks do not verify that claims were derived.Models average 62% of rationale points versus 74% of remaining L2 points; GPT-5.5 earns 50% of rationale credit and 90% of the rest.Source: §F.2 · p. 17

G Scoring Reliability Analysis; H Complete Per-Model Results

• Versioned deterministic verifiers bind quantities to subquestions, units, and signal identities; reports require all 40 expected samples, and Table 5 presents level scores for all 14 models.Stored results include scorer versions and task provenance fingerprints to prevent stale or partial results from entering reported aggregates.Source: G Scoring Reliability Analysis · p. 17, H Complete Per-Model Results · p. 17, Table 5 · p. 17

G.1 Zero-Pass Criterion Audit

• Six criteria are never scored perfectly by any model across 560 attempts, all at L5; every attempt at recovering 32 symbols scored zero.An independent decoder recovered the stream perfectly on 34 of 40 problems, supporting the audit’s conclusion that the task is solvable. The modulation inventory averaged 60% partial credit, while two extraction-filter criteria check constraints rather than stored answers.Source: §G.1 · p. 17

I Results by Question Type

• The three weakest types are L5 spectrum survey (15), coexistence analysis (23), and design (25) points; Shannon capacity is weakest at L4 with 43.Scores are points out of 100 averaged over models. Absolute power in dBm is the only weak L1 type at 67, and L2 energy and PSD is its level’s weakest type at 54.Source: I Results by Question Type · p. 18

L1: Signal-identity binding failure

• 9.3/20 on Q1 and 5.3/20 on Q2 follow when Llama 3.3 70B binds the chirp to −9.0 MHz instead of −7.41 MHz.The misplaced signal also disrupts frequency-list slots, strongest and weakest selections, power difference, per-signal power and SNR estimates, and bandwidth category; GPT-5.5’s calibrated PSD code preserves the correct binding.Source: J Per-Level Failure Case Analysis · p. 18

L2: Procedural fluency without conceptual grounding

• Models can compute 3-dB, null-to-null, and 99%-power bandwidths correctly yet lose L2 comparison credit by choosing 3-dB bandwidth for spectrum allocation.The verifier awards 2 points for choosing occupied-power bandwidth and 1 more for stating its consequence—energy capture or adjacent-channel interference—near that choice.Source: J Per-Level Failure Case Analysis · p. 18

L3: Cascading chain errors

• A QPSK-to-BPSK misidentification on an L3 link-budget problem propagates through bit rate, required Eb/N0, link margin, and feasibility, yielding 0/100.The bit-rate estimate is halved, required Eb/N0 shifts by about 3 dB, and the error exceeds the ±3 dB tolerance at each of the five chained sub-questions.Source: J Per-Level Failure Case Analysis · p. 18

L4: Multi-entity binding collapse

• A correct set of L4 occupied bandwidths still leads to a wrong overlap verdict when the model swaps which signal occupies each frequency range.The overlap verdict gates 40% of the sub-question score and its error invalidates downstream interference calculations.Source: J Per-Level Failure Case Analysis · p. 18

L5: Catalog-gated scoring cascade

• On L5 system-design problems, missing one of six catalog signals can zero prerequisite-gated Q2 interference-analysis or Q3 system-design sub-questions.The example is a weak burst emitter 8 dB below the strongest signal; its missed identity gates 33-point sub-questions in Q2 or Q3.Source: J Per-Level Failure Case Analysis · p. 18

L5 extraction-filter deficit breakdown

• 527 of 560 L5 extraction-filter attempts score zero, and none earns full credit because reported metrics fail to match measured design behavior.All 527 zeros arise from perceptual limits: 35 submissions lack a schema-valid answer and 492 select the wrong target; in 427 of those, the target is absent from the inventory, including 390 cases where two emitters are merged. Of 19 evaluated designs, 15 omit constraints; all 19 misreport retained fraction, and the 4 feasible designs include 3 within 4 dB of optimal SIR.Source: J Per-Level Failure Case Analysis · p. 18

K Model Response Examples

• Three paired case studies, one per ReconPilot backbone, compare free-form and ReconPilot responses to the same problems and show higher scores for the structured-pipeline responses.The examples illustrate how the pipeline converts partially scored free-form attempts into substantially higher-scoring responses.Source: K Model Response Examples · p. 18
• Figure 7 contrasts GPT-5.5’s calibrated signal-processing code with Llama 3.3 70B’s plausible but unverified answers, which deviate >10 dB from measured ground truth.The comparison uses excerpts from the same L1 problem, EMRB_L1_5000 (Q1–Q2).Source: Figure 7 · p. 19, K Model Response Examples · p. 19
• On the L5 example, analysis chains signal inventory into interference identification and filter design; it recovers six sources, identifies overlap 8/8, and finds 4 qualifying designs among 560 attempts.The filter passband satisfies the retention constraint; Figure 8 describes verification against ground truth and physical constraints.Source: K Model Response Examples · p. 19, Figure 8 · p. 19
• Despite scoring 70.0/100, the design retains 0.99 of the target at 1.5 dB post-filter SIR, not its reported 0.80 at 3.7 dB; symbol recovery earns 1/10 and the modem upgrade 0/8.Source: K Model Response Examples · p. 19

• Existing EM datasets use presegmented snippets or encoded representations with fixed classification outputs, while data-analysis benchmarks use structured tables with explicit schemas.Source: L Benchmark Comparison · p. 19, Table 6 · p. 20
• EMRB is the only benchmark described as providing raw, full-length I/Q recordings and requiring free-text engineering answers through self-directed code execution.Source: L Benchmark Comparison · p. 19
• On EMRB_L3_4037, a 2.9× root-symbol-rate error propagated through dependent questions in free-form execution, while ReconPilot’s region-map reconnaissance reached full credit on Q1, Q2, and Q5.Source: Figure 9 · p. 20