PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization

Lei Zhai, Zhihao Chang, Shuyuan Yang, Zhixi Feng

arXiv:2610.12142v1cs.AIeess.SP

TL;DR

Spectrum monitoring needs flexible analysis beyond predefined recognition and detection, but existing task-specific systems separate objectives. BATok adapts bounded signal tokens to I/Q observations, and experiments across two VLMs show gains in modulation recognition, positive-query detection, and grounding, with waveform evidence complementing waterfall images.

  • Problem

    Spectrum monitoring needs flexible detection, localization, and characterization beyond predefined-class recognition, while task-specific methods expose separate interfaces.

    Source: Introduction · p. 1

  • Method

    BATok uses length-adaptive, bounded signal tokens for VLM conditioning, while EMSpec-Instruct aligns I/Q signals, waterfall images, and language supervision across three tasks.

    Source: Introduction · p. 2

  • Results

    Across two VLMs, BATok improves modulation recognition and positive-query detection and grounding, with waveform evidence complementing waterfall images.

    Source: Conclusion · p. 8

  • Takeaways & Limitations

    Waveform evidence complements waterfall images in the reported spectrum-understanding results.

    Source: Conclusion · p. 8

    For Qwen3-VL-2B/LLaVA-1.5-7B, matched-box mIoU rises 0.49/0.58 points while empty-set accuracy falls 0.76/0.69 points, so grounding changes are non-uniform.

    Source: S3.4 Analysis of BATok Performance Gains · p. 19

Abstract

from arXiv · show

Electromagnetic spectrum monitoring increasingly requires flexible analysis beyond task-specific recognition and detection. Multimodal large language models offer a unified interface, but extending vision-language models (VLMs) to raw I/Q signals requires tokenization that balances fidelity against a strict budget. For signals, dense encoding causes token costs to grow with observation length, whereas fixed-resolution compression may discard short-duration or localized signal evidence. Thus, we propose \textbf{BATok}, a budget-adaptive signal tokenizer that adjusts token capacity to the input length while allocating that capacity according to the signal content. BATok constructs candidate representations from signal-derived features using lightweight multi-resolution branches, then combines a local energy prior with learnable queries to resample these representations into compact signal tokens. The number of tokens adapts to the input length while remaining strictly bounded. The resulting tokens are projected into the language embedding space of VLMs. We further introduce \textbf{EMSpec-Instruct}, a multimodal instruction dataset aligning raw I/Q signals, waterfall images, and language supervision for modulation recognition, structured detection, and language-conditioned signal grounding. Experiments show that BATok learns effective signal representations and achieves competitive performance across all tasks.

Introduction

Spectrum monitoring calls for flexible analysis beyond predefined recognition, but existing interfaces and encoders constrain how tasks and mixed-length I/Q observations are handled. The paper introduces BATok and EMSpec-Instruct to connect raw signals, waterfall images, and language supervision in a VLM framework.

Introduction

  • Monitoring must detect, localize, and characterize signals as well as recognize classes, while task-specific pipelines expose separate interfaces and detector–LLM answers remain constrained by upstream outputs.Supporting new queries can require manual rules, post-processing, or new model components.Source: Introduction · p. 1
  • Image-based RF interfaces may attenuate waveform-level amplitude, phase, and transient evidence, and their token capacity follows image resolution rather than the original observation.Generic audio and time-series tokenizers are not designed around the coupled physical structure of I/Q samples.Source: Introduction · pp. 1–2
  • Existing encoders face a mixed-length capacity mismatch: dense representations are costly for long signals, while uniformly small budgets may discard localized evidence.Their output granularity generally does not adapt jointly to input length and information distribution.Source: Introduction · p. 2
  • BATok uses signal-derived amplitude, power, and phase features with multi-resolution branches, then evidence-guided query resampling and a length-dependent, strictly bounded token budget.Its tokens are projected into the language-model embedding space and jointly processed with waterfall-image tokens and textual instructions.Source: Introduction · p. 2
  • EMSpec-Instruct aligns raw I/Q observations and waterfall images with language supervision for modulation recognition, structured detection, and language-conditioned signal grounding.The framework integrates BATok with a VLM through continuous signal-token projection; the paper reports waveform-level evidence complementary to waterfall images.Source: Introduction · p. 2

Related Work

Prior work learns useful waveform features and connects numerical or visual observations to language models, but often uses task-specific representations or token layouts not adapted to I/Q length and content. BATok instead provides a bounded continuous-token sequence whose capacity adapts to observation length and information distribution.

Representation Learning from Raw I/Q Signals

  • Raw-I/Q CNN, RNN, Transformer, and state-space encoders capture waveform evidence, but commonly produce global or fixed-granularity features for task-specific heads.BATok instead converts mixed-length I/Q observations into bounded continuous tokens for multimodal language models.Source: Representation Learning from Raw I/Q… · p. 2

Connecting Non-Textual Sequences to Language Models

  • Generic numerical-sequence interfaces do not explicitly preserve coupled I/Q evolution, while fixed-size patches grow in count with duration and fixed query sets give different-length observations equal capacity.BATok retains a bounded interface while adapting token count to valid length and allocating it with multi-resolution features, learned relevance, and an energy prior.Source: Connecting Non-Textual Sequences to… · p. 2

RF-Oriented Language and Multimodal Models

  • RF-oriented visual models reuse VLM interfaces for language-conditioned analysis, but inherit token layout and capacity from rendered image grids rather than numerical I/Q length and information distribution.The cited examples include modulation VQA and broader spectrum-understanding and waterfall-analysis systems.Source: RF-Oriented Language and Multimodal… · pp. 2–3

BATok: Budget-Adaptive Signal Tokenization

BATok adapts signal-token capacity to input length and distributes it over time according to signal content. It combines signal-derived features, multi-resolution encoding, and query-based resampling to produce compact tokens for VLM conditioning.

BATok: Budget-Adaptive Signal Tokenization

  • BATok sets a length-dependent, bounded token capacity and uses signal-aware resampling to distribute that capacity across the observation.Learned queries resample multi-resolution features using learned relevance and a local energy prior.Source: BATok: Budget-Adaptive Signal… · p. 3

Signal-Derived Feature Enhancement

  • The tokenizer preserves raw I/Q components while augmenting them with amplitude, power, and inter-sample phase features before a lightweight convolutional projection.These features provide complementary descriptions of envelope variation, signal activity, and local phase evolution.Source: Signal-Derived Feature Enhancement · p. 3

Multi-Resolution Temporal Encoding

  • Progressively coarser temporal branches trade localized waveform detail for broader context and stronger compression.The factorized temporal and channel mixing enriches each branch with linear sequence-length complexity rather than quadratic global self-attention cost.Source: Multi-Resolution Temporal Encoding · p. 3

Adaptive Token Budgeting and Resampling

  • A monotonic piecewise-constant budget assigns more tokens to longer signals while keeping the sequence compatible with the VLM context.Inputs within the same threshold interval share a token budget.Source: Adaptive Token Budgeting and Resampling · p. 4
  • Learned relevance and local energy softly modulate branch features before learnable-query cross-attention compresses each variable-length branch into bounded scale tokens.The modulation retains all valid temporal positions rather than removing low-scoring ones.Source: Adaptive Token Budgeting and Resampling · pp. 4–5

Multi-Resolution Token Aggregation

  • Scale-tagged branch tokens are concatenated, refined by a shared token-wise residual projection, and projected into the VLM language embedding space.The VLM models explicit cross-scale interactions after aggregation.Source: Multi-Resolution Token Aggregation · p. 5

VLM Integration and Training

Signal tokens are projected into the VLM language embedding space and combined with available image and instruction embeddings. The integrated model is trained to predict response tokens conditioned on optional signal, image, and instruction inputs.

VLM Integration and Training

  • BATok signal tokens are projected into the VLM language embedding space and supplied with image and instruction embeddings.Image embeddings come from the VLM visual encoder, while text embeddings come from its token embedding layer.Source: VLM Integration and Training · p. 5
  • The model supports image–text, signal–text, and image–signal–text input configurations.Signal, image, and instruction embeddings serve as conditioning context for the VLM.Source: VLM Integration and Training · p. 5
  • Supervised fine-tuning computes loss only on response tokens, using the signal, image, and instruction tokens as context.Training examples pair optional signal and waterfall-image inputs and textual instructions with target responses.Source: VLM Integration and Training · p. 5

EMSpec-Instruct Dataset

EMSpec-Instruct pairs raw I/Q observations, waterfall images, structured metadata, and natural-language queries to support instruction-conditioned spectrum tasks. It combines real and simulated single- and multi-signal data, with generated instructions and observation-level train–test separation.

EMSpec-Instruct Dataset

  • EMSpec-Instruct addresses the lack of existing electromagnetic dataset supervision for instruction-conditioned recognition, structured multi-signal detection, and language-conditioned grounding.It associates raw I/Q observations, waterfall images, structured signal metadata, and natural-language queries.Source: EMSpec-Instruct Dataset · p. 5
  • The dataset combines real and simulated single-signal data for modulation recognition with real and simulated multi-signal data for detection and grounding.Simulation expands modulation coverage and adds diverse acquisition lengths, signal densities, and time-frequency overlap patterns.Source: EMSpec-Instruct Dataset · p. 5
  • Metadata-derived task conditions guide instruction generation, with box answers for grounding retained only when they exactly match programmatically computed signals.The three task families are modulation recognition, structured detection, and language-conditioned grounding; Figure 3 summarizes construction and task and signal-length distributions.Source: EMSpec-Instruct Dataset · p. 6, Figure 3 · p. 6
  • The final dataset has 400,000 training and 67,883 test instructions, split by underlying observation before instruction generation to prevent observation-level overlap.Aligned multimodal and waterfall-only configurations support controlled comparison of image-only VLMs with BATok-equipped models.Source: EMSpec-Instruct Dataset · p. 6

Experiments

Across two VLM backbones, BATok-enhanced models improve modulation recognition and positive-query detection and grounding, while empty-set accuracy dips slightly. Standalone evaluations and ablations report competitive signal representations and support the tokenizer’s design choices.

Experimental Setup

  • VLM comparisons use image–text inputs for original models and aligned image–signal–text inputs for BATok-enhanced models on the same observations and partitions.Standalone evaluations and ablations use complete HisarMod2019 and EMSpec-Sim benchmarks; the VLM variants are fine-tuned for five epochs.Source: Experimental Setup · p. 6

Dataset-Level Task Feasibility

  • On HisarMod-Sub∪EMSpec-Sim-Sub, BATok reaches 49.36% Accuracy and 47.19% Macro-F1; YOLO26x reaches 0.9517 mIoU and 0.6925 F1@0.5 on structured detection.These task-specific results support the learnability of the recognition and detection subsets assembled from existing and simulated observations.Source: Dataset-Level Task Feasibility · pp. 6–7

Unified Instruction-Conditioned Spectrum Understanding

  • Modulation-recognition OA rises by 7.01 points for Qwen3-VL-2B and 18.56 points for LLaVA-1.5-7B; positive-query detection and grounding metrics also improve, while Empty Acc. falls by 0.76 and 0.69 points, respectively.The comparison reflects both BATok integration and mixed-modality training; localization gains are smaller, and joint input provides the best balance across evaluated tasks.Source: Unified Instruction-Conditioned… · p. 6, Table 1 · p. 7

Qualitative Analysis of Detection and Grounding

  • On representative simulated examples, BATok resolves modulation ambiguity between overlapping transmissions and selects the longest-duration transmission in relational grounding.The image-only baseline assigns both detections to 16QAM and selects a different high-energy band, respectively; these examples illustrate, rather than replace, aggregate metrics.Source: Qualitative Analysis of Detection and… · p. 7, Figure 4 · p. 8

Standalone Evaluation of Signal Representations

  • Across two benchmarks, BATok has the highest mean Accuracy (58.64%) and Macro-F1 (58.54%), but DNCNet retains the highest Accuracy on EMSpec-Sim.BATok’s average gains over the second-best method are 0.46 Accuracy and 1.65 Macro-F1 points.Source: Table 2 · p. 7, Standalone Evaluation of Signal… · p. 7

Ablation Studies

  • The piecewise K(T) schedule achieves the strongest mean Macro-F1 (58.54%) and stays within 0.06 points of Linear-high’s best mean Accuracy.Signal-derived features add 2.27/2.24 Accuracy/Macro-F1 points over I/Q alone, the third temporal branch adds 2.58/2.46 points over two branches, and combined energy and learned relevance exceed the stronger single bias by 1.67/1.32 points.Source: Ablation Studies · pp. 7–8, Table 3 · p. 8

Conclusion

BATok maps mixed-length raw I/Q observations to bounded, content-aware tokens for multimodal language models. Across two VLMs it improves modulation recognition and positive-query detection and grounding, but open-set and cross-domain generalization remain unverified.

Conclusion

  • BATok maps mixed-length raw I/Q observations to bounded, content-aware tokens for MLLMs.Its design combines a length-dependent budget, multi-resolution encoding, and energy–relevance-guided resampling.Source: Conclusion · p. 8
  • Across two VLMs, BATok improves modulation recognition and positive-query detection and grounding, with waveform evidence complementing waterfall images.Standalone results and ablations also support its representation and design.Source: Conclusion · p. 8
  • Open-set and cross-domain generalization remain unverified, and multi-resolution resampling gives BATok higher standalone latency than compact classifiers.The conclusion notes this despite BATok’s small parameter footprint and bounded output.Source: Conclusion · p. 8

S1 Architectural and Implementation Details of BATok

BATok turns variable-length I/Q observations into bounded signal-token sequences for VLM integration or standalone classification. Its architecture combines signal-derived features, multi-resolution temporal encoding, and length-adaptive resampling with backbone-specific integration.

S1.1 Input Representation and Normalization

  • Valid lengths guide branch masks and padded-token removal after signals are zero-padded to the minibatch maximum.Each input is represented as two I/Q channels and normalized by mean complex-sample power.Source: S1.1 Input Representation and… · p. 11

S1.2 Feature Lifting and Multi-Resolution Encoding

  • Five signal-derived channels feed a convolutional stem and three parallel temporal branches that capture features at multiple resolutions.The phase-difference feature avoids explicit phase unwrapping; each branch uses depthwise-separable convolution and a LiteMixer block in VLM experiments.Source: S1.2 Feature Lifting and… · p. 11, Table S1 · p. 11

S1.3 Length-Adaptive Budget and Scale Allocation

  • A monotonic, piecewise-constant schedule adapts the bounded token budget to input length, assigning one quarter each to small and mid branches and the remainder to the large branch.For K = 48, the branch budgets are (12, 12, 24); the two primary standalone benchmarks activate 32 or 48 tokens.Source: S1.3 Length-Adaptive Budget and Scale… · pp. 11–12, Table S2 · p. 12

S1.4 Placeholder Expansion and Modality Handling

  • At the <signal> anchor, projected signal tokens replace embeddings while the integration layer updates attention masks, positions, and labels and excludes signal-token labels from the language-model loss.Consistency checks require an I/Q input and anchor to appear together, while per-sample indexing supports mixed-modality minibatches.Source: S1.4 Placeholder Expansion and… · p. 12

S1.5 Backbone-Specific Integration

  • Both VLM backbones use trainable BATok and signal projectors with LoRA-adapted language layers, but use backbone-specific position handling and projector configurations.Qwen3-VL uses native multimodal rotary positions; LLaVA uses learned one-dimensional signal positions and a newly constructed projector.Source: S1.5 Backbone-Specific Integration · p. 12, Table S3 · p. 12

S1.6 Standalone Classification Head

  • For standalone classification, BATok tokens are masked-mean pooled and passed through a classification head trained with cross-entropy and optional label smoothing.BATok and standalone classifiers are trained for 100 epochs.Source: S1.6 Standalone Classification Head · p. 13

S2 EMSpec-Instruct Construction Details

EMSpec-Instruct is built around observations that pair I/Q signals, rendered waterfall images, and structured signal instances. Instructions and targets are derived from fixed metadata specifications, with observation-level partitioning maintained throughout construction.

S2.1 Source Observations and Partitioning

  • Separate single-signal and multi-signal streams supply recognition versus detection and grounding examples, and observations remain in their source partitions before and after instruction generation.Simulated single-signal data span 34 classes; multi-signal scenes use controlled overlap configurations and retain generation parameters as instance metadata for boxes.Source: S2.1 Source Observations and… · p. 13, S2 EMSpec-Instruct Construction Details · p. 13

S2.2 Instance Metadata

  • Each signal instance is encoded by its canonical class, time and frequency intervals, normalized box, and optional attributes; class aliases are canonicalized.Optional attributes include SNR, signal family, modulation, or communication standard.Source: S2.2 Instance Metadata · p. 14

S2.3 Time–Frequency Box Conversion

  • Boxes use normalized time–frequency coordinates, with frequency inverted to match the image convention; rounded coordinates are clamped, sorted, and invalid boxes rejected.Prompts give the physical observation ranges and define the axis directions and box ordering so models need not infer units from normalized coordinates.Source: S2.3 Time–Frequency Box Conversion · p. 14

S2.4 Task Construction

  • A deterministic semantic specification fixes each task, condition, matched instances, target boxes, and negative-query status before natural-language realization.Grounding conditions include class, interval, class–time–frequency conjunctions, duration or bandwidth relations, waterfall strength, and overlap; negative queries have empty targets.Source: S2.4 Task Construction · pp. 14–15

S2.5 Metadata-Conditioned Instruction Realization

  • Qwen3.5-Max realizes instruction wording, but recognition and detection answers come directly from annotations, while grounding outputs are retained only when they match the computed targets exactly.Records must pass checks for item order, context, classes and ranges, box matches, negative answers, and JSON parsing.Source: S2.5 Metadata-Conditioned Instruction… · p. 15

S2.6 Compact Target Grammar

  • Compact target grammars serialize class lists for recognition, boxes and classes for detection, and boxes for grounding; negative grounding uses an empty box list.The grammar permits parsing classes, boxes, and box–class correspondence without another language model.Source: S2.6 Compact Target Grammar · p. 15

S2.7 Sampling, Modalities, and Final Statistics

  • The final dataset contains 400,000 training records, with 30% for recognition and 70% for multi-signal understanding, and aligned image-only and signal-containing configurations.The signal configuration includes joint image–signal, signal-only, and image-only records; training observations range from 1,024 to 15,000,000 complex samples after deduplication.Source: S2.7 Sampling, Modalities, and Final… · pp. 15–16, Table S6 · p. 16

S3 Extended Experimental Evaluation

The extended evaluation examines BATok-enhanced VLMs across input configurations and task families, alongside task-specific checks of EMSpec-Instruct supervision. Results show modality-dependent effects, with additional analyses separating task performance, detection gains, and grounding behavior.

S3.1 Evaluation Metrics

  • The evaluation separates recognition accuracy, structured detection and grounding localization, parse compliance, and empty-set handling using task-specific metrics.Detection matching is class-aware at IoU≥0.5, while grounding evaluates selected boxes without class prediction.Source: S3.1 Evaluation Metrics · pp. 16–17

S3.2 Spectrum Understanding Across Input Configurations

  • 64.12% to 64.27% detection F1 results when signal input is added to image–text input for Qwen3-VL-2B+BATok; joint input also raises recognition OA and grounding F1 over image–text input.Joint-input recognition OA is 40.91% and 44.14% for Qwen3-VL-2B and LLaVA-1.5-7B; grounding F1 rises by 1.22 and 0.52 percentage points, respectively.Source: Table S8 · p. 17, S3.2 Spectrum Understanding Across… · p. 17

S3.3 Validation of EMSpec-Instruct

  • The validation tests whether assembled observations, labels, and boxes support learnable recognition and detection, and whether the same data support a unified instruction-conditioned interface.Aligned raw I/Q and derived waterfall observations also permit comparing retained modulation-class evidence across modalities.Source: S3.3 Validation of EMSpec-Instruct · p. 17

Multi-Task Instruction Supervision

  • 83.00% grounding F1@0.5 is highest and 95.24% mIoU ties highest under the complete three-task mixture; relative to grounding-only training, the mixture gains 1.03 and 0.09 percentage points.All variants use Qwen3-VL-2B+BATok and the same grounding test set.Source: Multi-Task Instruction Supervision · p. 17, Table S9 · p. 18

Modulation-Recognition Learnability

  • 49.36% accuracy and 47.19% Macro-F1 are BATok’s results on the aligned recognition subsets, versus 32.43% accuracy and 34.33% Macro-F1 for the strongest waterfall-image result.BATok uses 1.047M parameters and has 5.443-ms latency; its dedicated classifier exceeds the two BATok-enhanced VLMs by 8.45 and 5.22 accuracy points.Source: Table S10 · p. 18, Modulation-Recognition Learnability · p. 18

Structured-Detection Learnability

S3.4 Analysis of BATok Performance Gains

  • BATok’s detection gains are backbone-dependent: predominantly semantic for Qwen3-VL-2B and jointly geometric–semantic for LLaVA-1.5-7B.For grounding, matched-box mIoU rises while empty-set accuracy falls for both backbones, so the changes are not uniform across localization behaviors.Source: Table S12 · p. 19, S3.4 Analysis of BATok Performance Gains · p. 19

S3.5 Fine-Grained Grounding Analysis

  • 90.28% is Qwen3-VL-2B+BATok’s highest positive-grounding F1 among its subtypes, on time-conditioned queries; both models maintain near-perfect parse rates.Across positive grounding subtypes, Qwen3-VL-2B+BATok exceeds its LLaVA counterpart in F1.Source: S3.5 Fine-Grained Grounding Analysis · p. 19

S4 Extended Ablation and Efficiency Analysis

The supplementary analysis evaluates BATok’s standalone classification performance, token-budget strategies, length-stratified behavior, and component contributions. Results favor a compact piecewise budget and show gains associated with each of three design components.

S4.1 Benchmark-Level Classification Performance and Efficiency

  • BATok performs best on HisarMod2019 and has the highest EMSpec-Sim Macro-F1, while DNCNet has the highest EMSpec-Sim Accuracy.BATok uses 1.044M parameters and has 5.509-ms standalone latency; the authors attribute its higher runtime to sequence-level computation, not parameter count alone.Source: Table S14 · p. 20, S4.1 Benchmark-Level Classification… · p. 20

S4.2 Comparison of Token-Budget Strategies

  • 58.64% Accuracy and 58.54% Macro-F1 with 40 average tokens make the piecewise schedule 17.5% lower-budget than Linear-mid, with decreases of 0.41 and 0.13 points.Against Fixed-48, it uses 16.7% fewer tokens while improving Accuracy and Macro-F1 by 0.67 and 0.94 points.Source: Table S15 · p. 21, S4.2 Comparison of Token-Budget… · p. 21

S4.3 Length-Stratified Comparison under Matched Token Budgets

  • 37.67% overall Accuracy for the piecewise schedule exceeds Fixed-39’s 37.31%, while Fixed-39 has higher overall Macro-F1, 36.46% versus 35.46%.At T = 8,192, 16,384, and 32,768, the piecewise schedule improves Accuracy by 0.71, 1.46, and 0.24 points, respectively, at a nearly matched average budget.Source: Table S16 · p. 21, S4.3 Length-Stratified Comparison… · p. 21

S4.4 Component-Level Ablation Analysis

  • The full configuration has the strongest two-benchmark mean in all three ablation groups; features, a third temporal branch, and combined resampling cues add 2.27/2.24, 2.58/2.46, and 1.67/1.32 Accuracy/Macro-F1 points.The reported gains are relative to I/Q alone, two branches, and the stronger single-bias variant, respectively.Source: Table S17 · p. 22, S4.4 Component-Level Ablation Analysis · p. 22

S5 Visualization Analysis

The visualizations compare paired model predictions across structured detection and language-conditioned grounding cases. They show image-based localization alongside signal-level semantic evidence, with BATok affecting target selection in the grounding examples.

S5 Visualization Analysis

  • The visualization analysis expands examples to Qwen3-VL-2B and LLaVA-1.5-7B across narrow-band, crowded, and condition-constrained cases; full-test-set metrics, not these examples, support performance claims.Paired predictions use the same prompt and observation, with and without BATok.Source: S5 Visualization Analysis · p. 22

S5.1 Structured Detection

  • In two detection scenes, the image pathway generally recovers transmission extents, while errors cluster in class assignments; BATok adds evidence for distinguishing modulation variants.The cases pair broadband WiFi with narrow LoRa, and show four overlapping transmissions from three signal families.Source: S5.1 Structured Detection · p. 22, Figure S1 · p. 22, Figure S2 · p. 23

S5.2 Language-Conditioned Grounding

Loading 2610.12142v1…