Source-linked AI summary
mm-Pose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNs
Arindam Sengupta, Feng Jin, Renyuan Zhang, Siyang Cao
TL;DR
Human skeletal pose estimation needs to operate despite lighting and weather limitations, while radar point clouds are sparse and difficult to use directly. mm-Pose combines a radar-to-image representation with a forked CNN to estimate 3-D joint positions in real time. In single-person tests across four motions, it reconstructed 17 usable joints with centimeter-scale localization errors and real-time inference, while performance varied by axis.
Problem
Vision sensors can fail under poor lighting, adverse weather, or occlusion, while radar point clouds complicate classification and lack abundant labeled datasets for skeletal estimation.
Method
mm-Pose projects 3-D mmWave radar reflections into a low-size RGB radar-to-image representation and uses a forked CNN to predict human joint positions in 3-D.
Results
The system reconstructed 17 joints after excluding 8 outliers, with average errors of 3.2 cm in depth, 2.7 cm in elevation, and 7.5 cm in azimuth, and was verified for real-time estimation.
Takeaways & Limitations
mm-Pose supports real-time skeletal pose estimation from mmWave radar for applications including traffic monitoring, autonomous vehicles, patient monitoring, and defense.
Takeaways & Limitations
Training used four movements in a single-human scenario, so predictions may be unreliable for substantially different motions such as crouching or bending.
Abstract
from arXiv · showhide
In this paper, mm-Pose, a novel approach to detect and track human skeletons in real-time using an mmWave radar, is proposed. To the best of the authors' knowledge, this is the first method to detect >15 distinct skeletal joints using mmWave radar reflection signals. The proposed method would find several applications in traffic monitoring systems, autonomous vehicles, patient monitoring systems and defense forces to detect and track human skeleton for effective and preventive decision making in real-time. The use of radar makes the system operationally robust to scene lighting and adverse weather conditions. The reflected radar point cloud in range, azimuth and elevation are first resolved and projected in Range-Azimuth and Range-Elevation planes. A novel low-size high-resolution radar-to-image representation is also presented, that overcomes the sparsity in traditional point cloud data and offers significant reduction in the subsequent machine learning architecture. The RGB channels were assigned with the normalized values of range, elevation/azimuth and the power level of the reflection signals for each of the points. A forked CNN architecture was used to predict the real-world position of the skeletal joints in 3-D space, using the radar-to-image representation. The proposed method was tested for a single human scenario for four primary motions, (i) Walking, (ii) Swinging left arm, (iii) Swinging right arm, and (iv) Swinging both arms to validate accurate predictions for motion in range, azimuth and elevation. The detailed methodology, implementation, challenges, and validation results are presented.
I. INTRODUCTION
The paper motivates radar-based skeletal pose estimation as a robust alternative to vision systems and introduces mm-Pose for real-time 3-D tracking of more than 15 human joints.
- Vision-based pose systems support applications including patient monitoring, pedestrian monitoring, autonomous vehicles, and defense decision-making.
- Poor lighting, adverse weather, and target occlusion can make vision-based sensors ineffective.
- Radar is robust to lighting and weather because it illuminates targets with its own signals, but reflection point clouds complicate classification and require labeled data.
- mm-Pose uses mmWave radars and CNNs to estimate and track human skeletons in real time.
- The authors describe mm-Pose as the first mmWave-radar method to estimate the real-world positions of more than 15 distinct human joints.
II. LITERATURE REVIEW
Prior work established camera-based and RF-based human tracking but left gaps in radar-based spatial skeletal estimation. The paper addresses this with 77 GHz mmWave sensing, radar signal processing, and CNN-based keypoint prediction.
- Vision-based pose estimation: Earlier vision approaches primarily estimated 2-D poses, while monocular cameras make depth extraction challenging.
- RF-based tracking: Micro-doppler radar studies characterized human behavior but did not provide subjects’ spatial locations.
- RF-based tracking: Wi-Fi wearable sensors indirectly represent humans and cannot distinguish body parts, limiting their suitability for pose estimation.
- mm-Pose: mm-Pose uses 77 GHz mmWave radar and a forked CNN to predict more than 15 keypoints and construct human skeletons in real time.
- Radar signal processing: The radar processing chain derives range from FMCW beat frequency, velocity from Doppler processing, and angle from TDM-MIMO direction-of-arrival estimation.
- Radar signal processing: TDM-MIMO combines two transmitting and four receiving antennas, treating signals from the second transmitter as virtual receiving-array samples.
B. Neural Networks
Neural networks transform inputs through weighted, biased, nonlinear processing stages. CNNs extend this approach for images by learning spatial representations with fewer parameters than equivalent fully connected networks.
- Neural-network fundamentals: Each neural-network node applies a weighted input, bias, and nonlinear activation to produce its output.The activation function enables estimation of complex nonlinear relationships.
- Neural-network fundamentals: Neural networks comprise input, hidden, and output stages, with hidden-layer structure treated as a hyper-parameter.Hidden layers transform inputs into higher-dimensional representations for classification or regression.
- Convolutional neural networks: CNNs empirically perform better than MLPs for image tasks because convolutional layers learn multiple spatial representations.A CNN layer’s depth determines the number of transformed representations it generates.
- Convolutional neural networks: CNNs reduce computational complexity by reusing kernel weights, requiring 3k^2 parameters instead of 3N^2 for a simple N × N × 3 example.With D representations, CNN parameters increase as D × (3k^2), whereas fully connected MLP parameters increase as (3N^2)^D.
A. Radar-To-Image Data Representation
The radar data are converted from 3-D reflection points into compact RGB images that preserve spatial coordinates and reflection power. Projection addresses the large size and sparsity of direct volumetric representations.
- Radar representation: mmWave radar processing resolves range, velocity, and angle, then converts reflections into real-world x, y, z positions.The radar data cube contains fast-time, slow-time, and channel dimensions.
- Radar representation: Reflection points can be represented as a 3-D heatmap by encoding reflection power as an additional feature.The power-weighted point cloud provides information about reflecting-surface strength beyond point location.
- Representation challenges: A direct 5 m × 5 m × 5 m representation at 5 cm resolution produces a 100 × 100 × 100 × 3 input with extreme sparsity.The example contains only 256 reflection points among 10^6 pixels, making the representation parameter-intensive and computationally inefficient.
- Proposed radar-to-image representation: The proposed projection maps reflections onto XY and XZ planes, producing two 16 × 16 × 3 RGB images encoding coordinates and normalized power.Undetected pixels receive (0,0,0), substantially reducing CNN input size and computational complexity.
B. CNN Architecture
mm-Pose uses two projected radar images in separate CNN branches, combines their learned representations, and regresses 3-D coordinates for 25 skeletal joints. Reflection power and single-axis radar compatibility extend the architecture’s utility.
- CNN architecture: The forked CNN processes projected radar images in separate branches before concatenating their outputs for downstream MLP regression.CNN weight sharing reduces trainable parameters relative to an equivalent fully connected MLP.
- MLP regression: The concatenated N × N × 128 tensor is flattened and passed through a 3-layer MLP with 512, 256, and 128 nodes.The MLP uses 30% dropout and ReLU activation.
- MLP regression: The output layer contains 75 linear nodes representing X, Y, and Z coordinates for 25 skeletal joints, trained by minimizing MSE against ground truth.The model uses the Adam optimizer with gradient descent.
- Design advantages: The architecture can use two 90°-offset single-axis radars, avoids data association and complex 4D CNNs, and incorporates power to distinguish body-part reflections.The power feature can differentiate larger-RCS regions such as the torso from smaller-RCS regions such as the elbow.
A. Experimental Setup and Frames Association
The study combines two rotated mmWave radars with synchronized Kinect skeletal ground truth and collects four single-person motions. Radar frames are aligned with Kinect returns and converted into normalized paired images for model processing.
- Radar hardware and geometry: Two AWR 1642 radars are used, with the second rotated 90° so its azimuth measurements correspond to elevation.Each radar transmits a 3.072 GHz chirp centered at 79 GHz every 92 µs.
- Ground-truth acquisition: A synchronized Kinect supplies depth, azimuth, elevation, and UTC timestamps for 25 ground-truth joint positions.The radar and Kinect computers use a common time server, with clock slew on the order of one millisecond.
- Motion protocol: Two subjects perform walking, left-arm swing, right-arm swing, and both-arms swing in an open-space experiment.Subjects perform the actions one at a time in contiguous sets.
- Dataset: The dataset contains approximately 32,000 training, 6,000 validation/development, and 1,700 test samples.Test actions were performed in no ordered fashion for added robustness.
- Frame association and preprocessing: Radar frames are separated using frame-header module information, matched frame-by-frame to Kinect returns by UTC timestamps, and normalized across spatial coordinates and intensity.The normalized data generate two RGB images per frame corresponding to the two radar views.
B. Training the Architecture
The architecture uses a forked CNN trained on normalized radar-derived inputs, with validation-based checkpointing and dropout to support efficient learning and reduce overfitting.
- B. Training the Architecture: The forked CNN was selected over a complete MLP to reduce computational complexity and support real-time implementation.The CNN processes radar-derived representations rather than learning through a fully connected network alone.
- B. Training the Architecture: Training used normalized experiment data, the Keras Functional API, TensorFlow, an Adam optimizer, and mean-squared-error loss against ground truth.
- B. Training the Architecture: The test visualizations compare predicted and ground-truth 17-point skeletons across walking and three arm-swing motions.
- B. Training the Architecture: Validation-loss monitoring retained the best model parameters before overfitting, while dropout was used as a regularization technique.
C. Test Results and Analysis
Test evaluation measured joint-location errors across three spatial dimensions and used an average-joint-location model as the comparison baseline.
- C. Test Results and Analysis: The model’s test performance was measured using mean absolute error in depth, azimuth, and elevation for all 25 points.
- C. Test Results and Analysis: The baseline always outputs each joint’s average ground-truth location computed from the training data.
1) Localization Accuracy:
Localization analysis excluded eight hand-related outliers and evaluated the remaining 17 joints, showing centimeter-scale errors and lower errors than the baseline across the testing frames.
- 1) Localization Accuracy:: Eight hand-related outlier joints were excluded because their small radar cross sections produced consistently high errors, while the remaining 17 preserved skeletal interpretability.
- 1) Localization Accuracy:: The cumulative error distribution showed faster convergence for mm-Pose than the baseline, indicating a higher probability of lower localization error.
- 1) Localization Accuracy:: 3.2 cm in depth (X), 2.7 cm in elevation (Z), and 7.5 cm in azimuth (Y) were the reported average localization errors.
- 1) Localization Accuracy:: The proposed model outperformed RF-Pose3D in depth and elevation but had greater azimuth error, with 7.5 cm versus 4.9 cm.
- 2) Architecture Metrics:: A 3×3×3 kernel with 27 trainable parameters transforms 16×16×3 radar images, and average inference time was approximately 150 µs per frame.
- 1) Localization Accuracy:: mm-Pose’s localization error was an order of magnitude lower than the baseline model across the testing data.
D. Practical Implementation
The system was implemented as a ROS pipeline for real-time radar-based pose estimation, but its current robustness is bounded by limited motion diversity and costly data collection.
- D. Practical Implementation: The ROS implementation used sequential nodes to publish radar point clouds, encode 16×16×3 RGB radar images, and perform pose estimation.
- D. Practical Implementation: Training data covered four motions, so predictions may be unreliable for different spatial motions such as crouching or bending over.
- D. Practical Implementation: The radar-skeletal database was unavailable, making data acquisition the most expensive stage of the study.
- D. Practical Implementation: The representation projected 3-D radar point clouds into planes and encoded position and intensity in RGB images before forked-CNN prediction.
- D. Practical Implementation: The system was verified for real-time estimation using mmWave radars and the mm-Pose architecture on ROS.