PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Adversarial Evasion Attacks in the Context of Wireless Communications

Bryse Flowers, R. Michael Buehrer, William C. Headley

arXiv:1903.01563v1eess.SPcs.LGstat.ML

TL;DR

The paper examines whether RFML systems using raw-IQ samples are vulnerable to adversarial examples and how wireless attack conditions affect evaluation. It develops an RFML threat model and BER-oriented methodology, then applies FGSM to direct-access and OTA AMC attacks. The results show vulnerability even in OTA attacks, while receiver effects can substantially impair adversarial evasion.

  • Problem

    RFML adversarial threats require evaluation across distinct attack launch points and against BER rather than human perception.

  • Method

    The work develops an RFML threat model and BER-based evaluation methodology, using FGSM against raw-IQ automatic modulation classification in direct-access and OTA settings.

  • Results

    FGSM attacks degrade RFML classification more effectively than Gaussian noise in direct-access tests and also expose vulnerabilities in OTA attacks.

  • Takeaways & Limitations

    RFML adversarial examples are a credible threat, but receiver effects and communication constraints must be considered when assessing evasion.

  • Takeaways & Limitations

    OTA receiver effects can degrade adversarial evasion and may require sacrificing spectral efficiency or increasing transmission power to maintain the same BER.

Abstract

from arXiv · show

Recent advancements in radio frequency machine learning (RFML) have demonstrated the use of raw in-phase and quadrature (IQ) samples for multiple spectrum sensing tasks. Yet, deep learning techniques have been shown, in other applications, to be vulnerable to adversarial machine learning (ML) techniques, which seek to craft small perturbations that are added to the input to cause a misclassification. The current work differentiates the threats that adversarial ML poses to RFML systems based on where the attack is executed from: direct access to classifier input, synchronously transmitted over the air (OTA), or asynchronously transmitted from a separate device. Additionally, the current work develops a methodology for evaluating adversarial success in the context of wireless communications, where the primary metric of interest is bit error rate and not human perception, as is the case in image recognition. The methodology is demonstrated using the well known Fast Gradient Sign Method to evaluate the vulnerabilities of raw IQ based Automatic Modulation Classification and concludes RFML is vulnerable to adversarial examples, even in OTA attacks. However, RFML domain specific receiver effects, which would be encountered in an OTA attack, can present significant impairments to adversarial evasion.

I. INTRODUCTION

RFML adversarial-ML research addresses threats that differ from computer-vision attacks because wireless perturbations can be launched at different points and must be evaluated using BER. This work consolidates an RFML threat model and evaluates FGSM attacks against raw-IQ AMC, including OTA settings.

  • RFML uses raw IQ samples for spectrum-sensing tasks, but adversarial-ML vulnerabilities in this domain remain comparatively understudied.
  • RFML attacks differ by launch point: direct classifier access, synchronous transmission from the transmitter, or asynchronous transmission from a separate device.
  • Wireless adversarial success should be evaluated using bit error rate rather than human perception, the primary metric in image-recognition settings.
  • The work proposes a consolidated RFML threat model and uses FGSM to study raw-IQ AMC vulnerabilities in direct-access and OTA attacks.
  • Prior AMC work used FGSM and UAP variants with energy-ratio constraints, but did not model channel effects for OTA perturbations.

III. THREAT MODEL FOR RFML

The RFML threat model extends adversarial-ML categories to wireless-specific attack goals and capabilities. It distinguishes attacks by launch location and considers blind signal classification by an unsynchronized eavesdropper.

  • The work extends computer-vision threat models to capture adversarial goals and capabilities unique to RFML.
  • The system model considers an eavesdropper that detects, isolates, and modulation-classifies a signal with limited prior information and no synchronization.
  • Adversarial studies can support both vulnerability analysis for robust eavesdroppers and feasibility analysis for protecting transmissions from eavesdroppers.
  • The threat model distinguishes confidence reduction, untargeted misclassification, and targeted misclassification as adversarial goals.
  • Attacks are categorized by where the perturbation is launched, including direct access at the eavesdropper’s signal-processing chain.

2) Self Protect:

Self-protect attacks add perturbations at the transmitter, preserving synchrony but exposing them to receiver effects and affecting the intended receiver. RFML also includes asynchronous separate-emitter attacks, while signal-processing hardware can impair adversarial perturbations.

  • Self Protect: Self-protect attacks add the perturbation at the transmitter, where it propagates synchronously with the signal and affects the intended receiver.
  • Self Protect: Separate-emitter attacks combine the perturbation with the transmission only at the eavesdropping device and cannot rely on transmitter-perturbation synchronization.
  • Self Protect: RF propagation capabilities include directing energy with smart antennas to alter the signal-to-noise ratio observed by an eavesdropper.
  • Self Protect: RF front ends, nonlinear power amplifiers, and limited analog-to-digital precision can filter, distort, or discretize adversarial perturbations.
  • Self Protect: FGSM generates a single-step untargeted adversarial example constrained by a feature-space distance ϵ.
  • Self Protect: Wireless evaluation reformulates ϵ as a power-ratio bound and uses bit error rate rather than human perception as the primary metric.

A. Adapting FGSM

The methodology adapts FGSM perturbations to wireless signals by computing their energy and scaling them to a desired transmission-to-jamming power ratio. This constraint supports system trade-off evaluation but leaves joint bit-error-rate optimization for future work.

  • Energy normalization: The average energy per symbol is computed from the signal samples, with the modulated signal energy set to 1 for deriving the power ratio.The formulation uses samples per symbol, total samples, and individual time-domain samples.
  • Terminology: Jamming and perturbation signal are used interchangeably because the crafted electronic signal deliberately impairs an eavesdropper’s operation.
  • Complex perturbations: The complex FGSM gradient sign produces sample values of (±1, ±1j), enabling calculation of the jamming signal’s per-sample magnitude.
  • Power-ratio control: A closed-form scaling factor ϵ adjusts FGSM perturbations to achieve the desired Es/Ej energy ratio before adversarial examples are created.The scaled perturbation is added to the input signal under the selected power constraint.
  • Trade-off and scope: The fixed power-ratio constraint evaluates transmitter design trade-offs, but current attacks cannot reuse jamming energy for the underlying transmission.Future attacks could incorporate bit error rate so the energy serves both purposes, but that exploration is deferred.

B. Simulation Environment

The simulation environment combines digital signal processing and machine-learning frameworks to evaluate wireless adversarial attacks. GNU Radio implements the signal-processing flow graphs, while PyTorch supports machine learning, although the methodology is framework-independent.

  • Frameworks: The evaluation interfaces GNU Radio for DSP with PyTorch for machine learning in a simulation environment designed for wireless communications.The authors state that the methodology does not depend on these specific frameworks.

1) Modulation:

The modulation experiment generates labeled signals, crafts FGSM perturbations on DNN-sized slices, and evaluates them through a channel and demodulation pipeline. Classifier success is measured by Top-1 accuracy, with lower accuracy indicating stronger untargeted evasion.

  • Signal generation: The simulation uses BPSK, QPSK, 8PSK, and QAM16 with 8 samples per symbol, an 8-symbol filter span, roll-off factor 0.35, and 1000 examples per scheme.
  • Adversarial-signal construction: Signals are sliced into DNN-sized examples, dithered, processed with FGSM, concatenated into a jamming signal, and scaled across Es/Ej from 0 to 20 dB in 4 dB steps.
  • Channel model: The channel model includes Additive White Gaussian Noise and center-frequency offsets, with noise variance determined by the desired Es/N0.
  • Demodulation: Demodulation applies matched filtering, downsamples to one sample per symbol, and decodes symbols into bits while assuming symbol and frame synchronization.
  • Evaluation metric: Top-1 accuracy evaluates the classifier, and lower Top-1 accuracy defines greater success for untargeted adversarial attacks.

C. Automatic Modulation Classification Target Network

The target network is a convolutional raw-IQ modulation classifier evaluated on open-source and newly generated synthetic datasets. The experiments vary input size and assess accuracy across signal-to-noise conditions and direct-access attacks.

  • Target architecture: The target DNN uses two convolutional layers followed by two fully connected layers and accepts IQ samples as a [1, 2, N] tensor.The architecture uses extended 7-tap filters padded with three zeros on each side.
  • Dataset A: RML2016.10A contains 11 modulation types generated through a dynamic channel model at SNRs from -20 dB to 18 dB.
  • Dataset limitations: Dataset A provides only 128 complex samples per input and has limited center-frequency offsets, motivating an additional dataset for the complete evaluation.
  • Dataset B: The additional synthetic datasets use input sizes 128, 256, and 512 across five modulation schemes, with controlled filter and channel parameters.
  • Training: Training with Dataset A uses dropout, whereas Dataset B uses batch normalization to improve stability for larger examples; models are trained with Adam.
  • Accuracy versus input size: Increasing input size increases test accuracy across the studied SNR range, according to the Dataset B evaluation.
  • Training results: The direct-access training results compare FGSM perturbations with Gaussian noise at the same power ratio using overall classification accuracy.

V. ANALYSIS OF DIRECT ACCESS ATTACKS

Direct-access FGSM attacks substantially degrade raw-IQ AMC, while input size changes the classifier’s relative robustness as adversarial energy increases. The analysis also introduces difference in logits to track classification and output variation at the individual-example level.

  • Attack effectiveness versus Es/Ej: The direct-access experiment assumes adversarial examples receive no added noise and evaluates average classification accuracy against Es/Ej.The analysis repeats this comparison across models trained on Dataset B with different input sizes.
  • Attack effectiveness versus Es/Ej: At 10 dB, FGSM degrades classification below random guessing, an 8 dB improvement over Gaussian-noise degradation.At 30 dB, FGSM remains more effective than AWGN.
  • Attack effectiveness versus NN input size: At Es/Ej below 15 dB, the input-size accuracy ranking inverts: the smallest model is most accurate and the largest is least accurate.At very high Es/Ej, the largest-input model is most accurate; the second-largest surpasses it below 55 dB (ϵ ≈0.00044).
  • Difference in logits: Individual-example analysis sweeps Es/Ej from 40 to 0 dB in 1 dB steps to examine fine-grained classifier behavior.The study analyzes one example from each source modulation and visualizes DNN outputs before softmax.
  • Difference in logits: The BPSK perturbation is more distinguishable in Q samples than I samples, with differences most apparent near symbol locations.The perturbation’s box shape results from the sign operation in the FGSM construction.
  • Difference in logits: Difference in logits subtracts the largest incorrect-class output from the true-class output, jointly capturing classifier accuracy and output variance.Positive values indicate correct classification, whereas negative values indicate untargeted adversarial success.

2) Classifier Output versus Es/Ej:

Classifier outputs respond differently to adversarial energy across modulation examples, and AWGN can impair adversarial success. These results motivate evaluating stochastic-input attacks separately from deterministic direct-access attacks.

  • Classifier Output versus Es/Ej: At Es/Ej of 10 dB, the BPSK example is untargetedly misclassified, but the classifier still identifies its linearly modulated signal family.The predicted modulation order increases as Es/Ej increases, while negative difference in logits indicates misclassification.
  • Classifier Output versus Es/Ej: The QAM16 example reaches untargeted misclassification below 30 dB, whereas the BPSK example requires below 15 dB.At very low Es/Ej, the QAM16 example remains classified as QAM, with the model narrowly predicting QAM64.
  • Mutation Testing with AWGN: For a BPSK example at Es/Ej of 10 dB, an Es/N0 of 17 dB makes the 75th percentile of difference in logits positive in some trials.The experiment varies Es/N0 from 20 to 0 dB with 1000 trials at each SNR.
  • Mutation Testing with AWGN: AWGN sensitivity is most pronounced when adversarial perturbation and noise have similar power, rather than across all adversarial examples.The cited results qualify broader claims that adversarial examples are generally more noise-sensitive.
  • Mutation Testing with AWGN: Noise can reduce adversarial success, so stochastic-input attacks require different evaluation from deterministic direct-access attacks.The section concludes that raw-IQ AMC is vulnerable to untargeted adversarial examples while receiver noise can impair attack effectiveness.

VI. ANALYSIS OF SELF PROTECT ATTACKS

Self-protect attacks must preserve reliable transmission while reducing an eavesdropper’s classification accuracy. Across tested modulations, adversarial perturbations could lower classification for a given BER, but receiver noise and attack intensity created important trade-offs.

  • Threat model and evaluation: AWGN, sample timing, and frequency offsets are receiver effects that can impair OTA adversarial success.Self-protect attacks must balance evasion against transmitting information across the wireless channel.
  • BPSK trade-offs: Classification accuracy can fall to ≈0% with no noticeable BER change for BPSK at Es/Ej of 4 dB, but only above 15 dB SNR.At the 10 dB baseline, Es/Ej of 8 dB requires an ≈2 dB SNR increase to maintain BER.
  • QPSK trade-offs: A 4 dB SNR increase maintains QPSK BER while reducing classification accuracy to < 20%.This illustrates the transmission-power cost of preserving BER during adversarial evasion.
  • Higher-order modulations: For higher-order modulations, attacks with Es/Ej ≤8 dB already produce bit errors, while QAM16 attacks can harm the receiver more than the eavesdropper.QAM16 BER reaches ≈16% at Es/Ej of 4 dB and ≈25% at 0 dB even before added noise.
  • Noise effects: AWGN reduced adversarial success in 3 out of 4 tested source modulations, making attacks most effective at high SNRs.For BPSK at Es/Ej of 8 dB, an eavesdropper at Es/N0 of 10 dB still classifies the signal correctly 20% of the time.

B. Frequency Offset

Center-frequency offsets can impair adversarial examples even when the baseline classifier generalizes within its training range. Small estimation errors therefore need to be considered when generating OTA attacks.

  • Experimental setup: Dataset B trains over normalized frequency offsets of ±1%, while the evaluation sweeps offsets beyond that range.The experiment uses input size 128 to compare with Dataset A results.
  • Results: Adversarial examples achieve ≈10% higher classification accuracy than the baseline even at offsets of ±0.1%.The effect appears at both 20 and 10 dB SNR.
  • Implication: Minute frequency-offset estimation errors can negatively affect adversarial machine learning despite baseline generalization within the training range.Frequency-offset robustness must therefore be included in adversarial generation methods.

C. Time Offset

Sample-time offsets have limited impact on the baseline classifier but can substantially weaken adversarial examples. OTA evaluations must therefore account for timing misalignment and receiver-specific variation.

  • Experimental setup: Sample-time offsets model misalignment between adversarial perturbation crafting and signal-classification windows.The experiment shifts the slicing start index across the 128-sample input at 10 and 20 dB SNR.
  • Results: A four-sample time-window shift increases classification accuracy by 20% for attacks with Es/Ej of 12 dB.Offsets beyond eight samples add no significant impairment beyond the symbol period.
  • Implications: Receiver effects can degrade OTA evasion, so future attacks must generalize over them while limiting impact on the underlying transmission.The conclusion identifies AWGN, sample-time offsets, and center-frequency offsets as relevant effects.
  • Future defenses: Multiple signal views may help detect or correctly classify adversarial examples, but extra inferences can constrain real-time sensing bandwidth.Offline DNN training offers an alternative defense path under SWaP constraints.
  • Conclusion: Adversarial machine learning is a credible and evolving threat to RFML systems that warrants consideration in future research.The conclusion frames this as a continuing research concern rather than a resolved problem.
Loading 1903.01563v1…