PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Radar-Based Gesture Detection and Recognition Built in an Edge-Computing Platform

Yuliang Sun, Tai Fei, Xibo Li, Alexander Warnecke, Ernst Warsitz, Nils Pohl

arXiv:2005.10145v1eess.SP

TL;DR

Gesture recognition needs to avoid delay from fixed-window event detection when gesture durations vary, while remaining practical for edge devices. The system encodes hand profiles into a feature cube for a shallow CNN and detects gesture tails. It achieved strong reported F1-scores in hardware-in-the-loop tests and recognized gestures on an edge platform in approximately 33 ms.

  • Problem

    Detecting a gesture’s start and then waiting through a fixed number of measurement-cycles can delay classification when gesture durations vary.

    Source: A. Problem Statement · p. 4

  • Method

    A multi-feature encoder combines range, Doppler, azimuth, elevation, and temporal information in a feature cube for a shallow CNN, with an STA/LTA detector identifying gesture tails.

    Source: §I · p. 2, §VIII · pp. 8–9

  • Results

    In hardware-in-the-loop tests, the approach achieved 94.17% and 88.58% F1-scores using two taught and two untaught subjects as test sets, respectively.

    Source: §VIII · p. 9

  • Takeaways & Limitations

    The system recognizes 12 gestures in real time on an edge platform with limited memory and computational capability.

    Source: §I · p. 2

Abstract

from arXiv · show

In this paper, a real-time signal processing frame-work based on a 60 GHz frequency-modulated continuous wave (FMCW) radar system to recognize gestures is proposed. In order to improve the robustness of the radar-based gesture recognition system, the proposed framework extracts a comprehensive hand profile, including range, Doppler, azimuth and elevation, over multiple measurement-cycles and encodes them into a feature cube. Rather than feeding the range-Doppler spectrum sequence into a deep convolutional neural network (CNN) connected with recurrent neural networks, the proposed framework takes the aforementioned feature cube as input of a shallow CNN for gesture recognition to reduce the computational complexity. In addition, we develop a hand activity detection (HAD) algorithm to automatize the detection of gestures in real-time case. The proposed HAD can capture the time-stamp at which a gesture finishes and feeds the hand profile of all the relevant measurement-cycles before this time-stamp into the CNN with low latency. Since the proposed framework is able to detect and classify gestures at limited computational cost, it could be deployed in an edge-computing platform for real-time applications, whose performance is notedly inferior to a state-of-the-art personal computer. The experimental results show that the proposed framework has the capability of classifying 12 gestures in real-time with a high F1-score.

I. INTRODUCTION

Radar-based gesture recognition offers touchless interaction, but prior approaches either omit gesture cues such as azimuth, elevation, and temporal information or require computation and memory beyond typical embedded systems. This paper combines a compact multi-feature representation, gesture-tail detection, and a shallow CNN for real-time edge-platform recognition.

I. INTRODUCTION

  • Radar supports touchless interaction and is insensitive to ambient light, can penetrate dielectric materials, and offers privacy advantages over cameras.The introduction also notes that computer-vision hand-motion processing is usually not power-efficient for wearable and mobile devices.Source: §I · p. 1
  • Prior systems either omit some combination of range, Doppler, azimuth, elevation, and temporal cues or use computationally demanding neural networks unsuitable for many embedded systems.Some prior methods could not distinguish gestures with similar range and Doppler information.Source: §I · pp. 1–2
  • The framework recognizes 12 gestures in real time on an edge platform with limited memory and computational capability.Its contributions also include comparison with an offline benchmark.Source: §I · p. 2
  • The proposed multi-feature encoder combines range, Doppler, azimuth, elevation, and temporal information in a reduced-dimension feature cube for efficient processing.The feature cube is used as input to a shallow CNN rather than a deep CNN connected with recurrent networks.Source: §I · p. 2
  • An STA/LTA-based hand activity detector identifies the tail of a gesture so relevant preceding measurement-cycles can be assembled for classification.The approach is intended to avoid the delay associated with waiting a fixed number of cycles after detecting a gesture’s start.Source: §I · p. 2

II. SHORT-RANGE FMCW RADAR SYSTEM

The radar system uses a 60 GHz linear-chirp FMCW waveform to measure hand-scatter signals. Range and relative velocity determine the beat frequencies, while an L-shaped receive array supports azimuth and elevation estimation.

II. SHORT-RANGE FMCW RADAR SYSTEM

  • The system uses a linear chirp sequence frequency-modulation waveform, then mixes, filters, and samples the signal from hand point scatters at each measurement-cycle.The discrete beat signal is described for each of the z-th receive antennas.Source: §II · p. 2
  • Range and relative velocity determine the range and Doppler frequencies of each hand point scatter.The equations define these frequencies using range, velocity, bandwidth, chirp duration, wavelength, and the speed of light.Source: §II · p. 2
  • An L-shaped receive antenna array uses antenna spacing d = λ/2 to calculate azimuth and elevation angle of arrival.The array pair Rx1–Rx0 is used for elevation, and Rx1–Rx2 for azimuth.Source: §II · p. 2

III. MULTI-FEATURE ENCODER

The encoder applies a two-dimensional FFT to each measurement-cycle’s beat signal to obtain range-Doppler information. It uses windowing and incoherent antenna-spectrum summation in this signal-processing stage.

A. 2-D Finite Fourier Transform

  • A 2-D FFT is applied to each measurement-cycle’s discrete beat signal to extract range and Doppler information.The resulting complex-valued range-Doppler spectrum is calculated for each receive antenna.Source: A. 2-D Finite Fourier Transform · pp. 2–3
  • The range-Doppler spectrum uses a 2-D window function, with p and q indexing range and Doppler frequency, respectively.The stated frequency resolutions are Δfr = 1/Tc and ΔfD = 1/(IcTc).Source: A. 2-D Finite Fourier Transform · p. 3
  • The range-Doppler spectra from the three receive antennas are summed incoherently to improve signal-to-noise ratio.Source: A. 2-D Finite Fourier Transform · p. 3

B. Range-Doppler Estimation

The method selects the K strongest points in each range-Doppler spectrum to obtain range, Doppler, and magnitude information, then estimates azimuth and elevation from phase differences across receive antennas.

B. Range-Doppler Estimation

  • The K highest-magnitude points in each range-Doppler spectrum provide the hand's range, Doppler frequencies, and magnitudes.K is predefined; its choice is discussed in Section VII-A.Source: B. Range-Doppler Estimation · p. 3

C. Azimuth and Elevation Angle Estimation

  • Azimuth and elevation for each selected point are calculated from phase differences between corresponding complex-valued spectra from two receive antennas.The phase differences are taken at the same positions in the two receive-antenna spectra.Source: C. Azimuth and Elevation Angle… · p. 3

D. Feature Cube

The feature cube organizes five attributes of selected range-Doppler points over multiple measurement-cycles. The problem statement motivates detecting a gesture's tail rather than its start to reduce classifier delay.

D. Feature Cube

  • The feature cube V has dimensions IL×K×5, encoding range, Doppler, azimuth, elevation, and magnitude for K selected points over IL measurement-cycles.Its five channels correspond to those five attributes.Source: D. Feature Cube · p. 3, Figure 3 · p. 3

A. Problem Statement

  • Detecting gesture starts and then collecting IL measurement-cycles can delay classification because gesture durations vary and are usually shorter than IL.The multi-feature encoder requires IL measurement-cycles.Source: A. Problem Statement · p. 4, Figure 4 · p. 4
  • The proposed hand activity detector instead detects when a gesture finishes, allowing gesture data to be forwarded without the start-triggered delay.Hand activity detection also means the classifier is activated when a gesture is detected rather than on every measurement-cycle.Source: A. Problem Statement · pp. 3–4

B. STA/LTA-based Gesture Detector

The detector uses exponential moving averages over short and long windows to identify gesture tails. It first compensates the measured magnitude for range, then applies activity and movement-finish conditions.

B. STA/LTA-based Gesture Detector

  • Exponential moving averages of the range-weighted magnitude are computed over short and long windows, whose lengths are L1 and L2.The smoothing factor α is predefined.Source: B. STA/LTA-based Gesture Detector · p. 4
  • The detector uses a range-weighted magnitude, multiplying maximal range information by measured magnitude to form a coarse RCS estimate that partially compensates for path loss.The measured magnitude depends on path loss and radar cross section; the compensation is partial.Source: B. STA/LTA-based Gesture Detector · p. 4
  • A gesture tail is detected when the long-window average exceeds γ1 and the short-to-long-window mean EMA ratio falls below γ2, indicating activity followed by movement finishing.The thresholds β, γ1, and γ2 should be chosen for the application scenario.Source: B. STA/LTA-based Gesture Detector · p. 4

V. SUPERVISED LEARNING

The classifier applies a shallow CNN to the feature cube, with its design tuned for accuracy and limited edge-computing resources. The described evaluation setup uses an edge platform and a dataset of 7,200 gesture realizations.

V. SUPERVISED LEARNING

  • The feature-cube classifier uses four 3×3 convolutional layers with 64 kernels each, followed by fully connected layers and a softmax output.The first convolutional layer has depth five to match the input's range, Doppler, azimuth, elevation, and magnitude channels; dropout follows the first two fully connected layers.Source: §V · pp. 4–5
  • The network was tuned for classification accuracy, low resource use, and deployment on an edge platform; the framework was implemented on a Jetson Nano with TensorRT supporting the CNN.The multi-feature encoder and hand activity detector were implemented without runtime optimization.Source: §V · p. 5, §VI · p. 5

VI. SCENARIO AND GESTURE DATASET DESCRIPTION

  • The dataset contains 7,200 realizations: 12 gestures performed 30 times by each of 20 subjects, including both taught and self-interpreted performances.Subjects in the second group received only one example per gesture and performed gestures using their own interpretations.Source: §VI · p. 5
  • The classifier observes 40 measurement-cycles, totaling 1.36 seconds, to include hand movement because the dataset gestures take less than 1.2 seconds.Each measurement-cycle lasts 34 ms.Source: §VI · p. 5

VII. EXPERIMENTAL RESULTS

The evaluation compares offline performance with literature benchmarks using LOOCV and tests real-time capability online. It also selects K to balance classification accuracy and computational complexity.

VII. EXPERIMENTAL RESULTS

  • The evaluation compares offline benchmarks using LOOCV and assesses real-time performance with an online test.The real-time test feeds recorded radar data from four test subjects to the edge-computing platform.Source: §VII · p. 5

A. Determination of Parameter K

  • Mean average accuracy reaches approximately 95% at K = 25, after which increasing K barely improves accuracy.K = 25 yields a 40 × 25 × 5 feature cube, chosen to retain high accuracy with low computational complexity.Source: A. Determination of Parameter K · p. 5

B. Off-line Performance Evaluation

  • The offline comparison uses LOOCV to assess the proposed multi-feature encoder + CNN against four CNN/LSTM benchmarks, including AoA-augmented variants.The proposed model takes a 40 × 25 × 5 feature cube; the AoA-augmented benchmark inputs are 40 × 32 × 32 × 3.Source: B. Off-line Performance Evaluation · pp. 5–6

1) Classification Accuracy and Training Loss Curve:

The proposed multi-feature encoder + CNN achieves accuracy close to the AoA-augmented 3-D CNN + LSTM with lower computational cost and model size. It also maintains high accuracy on taught and untaught subjects.

1) Classification Accuracy and Training Loss Curve:

  • The proposed CNN’s training loss converges fastest among the three network structures and approaches zero at around training step 2,000.The authors attribute easier training to the feature cube’s compact form while retaining gesture characteristics.Source: 1) Classification Accuracy and… · p. 6, Figure 10 · p. 7

3) Computational Complexity and Memory:

2) Confusion Matrix:

  • Average accuracy is approximately 98.47% for taught subjects and 93.11% for untaught subjects using the proposed multi-feature encoder + CNN.The untaught-subject result reflects an approximately 5% accuracy degradation.Source: 2) Confusion Matrix · p. 7

C. Real-time Performance Evaluation

The real-time evaluation tests the system on taught and untaught subjects, including random motions as negative samples. The system achieves F1-scores of 94.17% and 88.58%, respectively, with recall reductions linked to hand-activity detection errors.

1) Precision, Recall and F1-score:

  • Real-time F1-score reaches 94.17% for taught subjects and 88.58% for untaught subjects.The evaluation includes recorded data from four test subjects and random motions as negative samples.Source: C. Real-time Performance Evaluation · p. 7, 1) Precision, Recall and F1-score · p. 8
  • Average precision and recall are 93.90% and 94.44% for taught test subjects, versus 91.20% and 86.11% for untaught test subjects.Each test subject contributes 60 realizations for each gesture.Source: 1) Precision, Recall and F1-score · pp. 7–8
  • Real-time recall falls by approximately 4% for taught and 7% for untaught subjects relative to offline results, with missed gestures and premature HAD triggers cited as causes.The authors mention missed detection of Pinch index and premature termination at the turning point of slow Cross gestures.Source: 1) Precision, Recall and F1-score · p. 8

2) Detection Matrix:

The system reports gesture-recognition results in offline and hardware-in-the-loop evaluations, alongside detection and runtime measurements. Its conclusion describes a feature-cube pipeline with a shallow CNN and STA/LTA-based gesture detection, while noting limits on subject diversity and radar placement.

VIII. CONCLUSION

  • The multi-feature encoder turns range, Doppler, azimuth, elevation, and temporal information into a feature cube for a shallow CNN; an STA/LTA detector detects gesture tails.The detector is proposed to reduce latency from the system’s fixed measurement-cycle requirement.Source: §VIII · p. 8

2) Detection Matrix:

  • Across 1,864 test samples, the system detected 1,388 true positives with 25 false alarms; FAR and MDR were 5.90% and 3.61%, respectively.The passage distinguishes HAD misses from classifier misses, where HAD detects a gesture but the classifier rejects it as an RM.Source: 2) Detection Matrix · p. 8, Table V · p. 9

3) Runtime:

  • Approx. 33 ms overall runtime comprised 7.12 ms for the multi-feature encoder, 0.38 ms for HAD, and 25.84 ms for the CNN.The encoder and HAD ran on the CPU using unoptimized C/C++ code; the CNN ran on the GPU using TensorRT.Source: 3) Runtime · p. 8, Table VI · p. 9

VIII. CONCLUSION

  • LOOCV accuracy was 98.47% for taught and 93.11% for untaught subjects; HIL F1-scores were 94.17% and 88.58%, with approx. 33 ms gesture recognition.The conclusion also reports that the trained shallow CNN has a small model size and requires few GFLOPs.Source: §VIII · p. 9
  • The authors identify diverse use-case-specific gesture datasets as future work and warn that classification accuracy might decrease when the radar moves relative to the user.They suggest ego-motion compensation as a possible response to the nonstationary-radar setting.Source: §VIII · p. 9
Loading 2005.10145v1…