PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of mmWave-based Human Sensing: Technology, Platform and Applications

Jia Zhang, Rui Xi, Yuan He, Yimiao Sun, Xiuzhen Guo, Weiguo Wang, Xin Na, Yunhao Liu, Zhenguo Shi, Tao Gu

arXiv:2308.03149v1cs.NI

TL;DR

mmWave human sensing lacks a comprehensive synthesis of its technology, platforms, applications, and challenges despite rapid development and broad use. This survey reviews hardware platforms and sensing techniques, organizes existing work into four sensing-granularity categories, and discusses future directions. It concludes that suitable pipeline techniques should be selected for each task, while hardware limitations and multitask complexity remain important boundaries.

  • Problem

    Existing mmWave human sensing works leave a significant gap in comprehensive coverage of their performance, challenges, limitations, and development trends.

  • Method

    The survey reviews hardware platforms, sensing techniques, applications, and existing works organized into four categories by sensing granularity.

  • Results

    The survey concludes that point clouds and deep learning often suit contour-related tasks, while phase waveforms and frequency-domain analysis suit micro-motion tasks.

  • Takeaways & Limitations

    Selecting techniques suited to each sensing task is recommended for obtaining better sensing results.

  • Takeaways & Limitations

    Current mmWave sensing remains constrained by insufficient antennas, limited transmission power, and limited hardware denoising capability.

Abstract

from arXiv · show

With the rapid development of the Internet of Things (IoT) and the rise of 5G communication networks and automatic driving, millimeter wave (mmWave) sensing is emerging and starts impacting our life and workspace. mmWave sensing can sense humans and objects in a contactless way, providing fine-grained sensing ability. In the past few years, many mmWave sensing techniques have been proposed and applied in various human sensing applications (e.g., human localization, gesture recognition, and vital monitoring). We discover the need of a comprehensive survey to summarize the technology, platforms and applications of mmWave-based human sensing. In this survey, we first present the mmWave hardware platforms and some key techniques of mmWave sensing. We then provide a comprehensive review of existing mmWave-based human sensing works. Specifically, we divide existing works into four categories according to the sensing granularity: human tracking and localization, motion recognition, biometric measurement and human imaging. Finally, we discuss the potential research challenges and present future directions in this area.

I. INTRODUCTION

mmWave human sensing offers contactless, fine-grained sensing, but human mobility, weak vital signals, and individual differences create distinctive challenges. This survey organizes the technology, platforms, applications, challenges, and future directions of the field.

  • Motivation: mmWave signals provide high sensing sensitivity, precision, and directional capabilities through high frequency, large bandwidth, short wavelength, and integrated antennas.These properties support beamforming and directional sensing.
  • Applications: mmWave human sensing supports tasks including tracking and localization, activity recognition, vital-sign monitoring, sound recovery, and human imaging.The survey describes applications spanning multiple sensing granularities.
  • Challenges: Human mobility, low-SNR vital signals, movement-related noise, physiological variation, and individual differences complicate accurate sensing.Vital signals can be affected by physiology, psychology, and the surrounding environment.
  • Survey scope: The survey addresses a gap left by prior surveys by comprehensively reviewing mmWave-based human sensing works and their key challenges.It focuses specifically on human sensing rather than only communication, applications, or general sensing techniques.
  • Contributions: The paper summarizes hardware platforms, datasets, and sensing-pipeline techniques, then categorizes works by sensing granularity and discusses challenges and future directions.Its taxonomy covers human tracking and localization, motion recognition, biometric measurement, and human imaging.

III. PLATFORM & DATASETS

The survey reviews mmWave hardware platforms and datasets, including FMCW radars and 60GHz probes, and explains how their integrated components capture sensing information. These platforms support applications ranging from localization and vital sensing to human imaging.

  • Platforms and datasets: mmWave sensing experiments use commercial FMCW radars, 60GHz probes, and customized devices, alongside public datasets for activity, vital, and 3D-pose sensing.The survey compares hardware products and lists datasets containing signals for different human statuses.
  • FMCW Radar: An mmWave radar system contains transmit/receive, radio-frequency, analog, and digital components that generate, transmit, mix, and process radar signals.The transmitter can use pulsed, FSK, CW, or FMCW waveforms.
  • FMCW Radar: Complex-signal mixing produces I and Q components with equal amplitude and frequency but a quarter-cycle phase shift.Single-channel I mixing instead provides the absolute frequency shift.
  • FMCW Radar: Commercial radar platforms integrate MMIC, AiP, SiGe, or RFCMOS technologies with processors and accelerators in compact form factors.TI IWR1843 integrates a 3TX, 4RX system, PLL, ADC, DSP, ARM processor, and hardware accelerator.
  • 60GHz Probe: 60GHz WiFi probes use commodity Qualcomm 802.11ad chipsets with 32 antennas and receiver-side Golay correlation to obtain channel impulse responses.These CIR measurements support human tracking and localization, vital sensing, and human imaging.

C. Customized Hardware

Customized mmWave platforms expand sensing flexibility through reconfigurable hardware, specialized antenna arrays, and integrated processing. The survey also catalogs open datasets spanning activities, rehabilitation, gestures, gait, and challenging environmental conditions.

  • Customized platforms: Customized platforms support reconfigurable RF front-ends, software-defined processing, phased arrays, and real-time configurable antenna systems.WiMi, OpenMili, M^3, and mm-FLEX illustrate progressively configurable hardware and processing designs.
  • Customized platforms: M^3 provides up to eight 32-element phased arrays at an order-of-magnitude lower cost than existing commercial mmWave software-defined radios.
  • Open datasets: Open datasets cover rehabilitation, activity recognition, gestures, gait, pose, and multimodal human sensing, using radar data with labels or complementary sensors.Examples include MARS, MMActivity, M-gesture, mmGait, and mRI.
  • Design considerations: Platform selection must match sensing requirements because carrier frequency, bandwidth, antenna arrangement, and transmission power determine sensing range, granularity, and dimensions.Vital sensing requires sub-millimeter accuracy, while human imaging requires two-dimensional spatial sensing capabilities.
  • Design considerations: Limited antenna counts can reduce angular resolution and point-cloud density, weakening robustness in practical FMCW radar applications.
  • Customized platforms: Customized hardware can add adjustable polarization, scene-adapted denoising, and task-specific antenna arrangements beyond typical COTS devices.These designs are intended to help researchers explore novel mmWave human-sensing techniques.

IV. KEY TECHNIQUES

The survey presents mmWave human sensing as a pipeline from signal capture through preprocessing and feature extraction to sensing models and tasks. For FMCW radar, successive transforms derive range, velocity, and angular position from reflected signals.

  • Pipeline overview: The general sensing pipeline comprises data capture, signal preprocessing, feature extraction, sensing models, and task-specific analysis.Raw received signals are sampled, converted into data, processed into signal forms, and denoised before sensing.
  • FMCW radar capture: FMCW radar uses chirps whose transmit–receive frequency difference reveals propagation time and object distance.The transmitted and received signals are mixed to obtain an intermediate-frequency beat signal.
  • FMCW radar capture: Range-FFT separates reflected signal components by range and produces range-spectrum information from within-chirp samples.
  • FMCW radar capture: Doppler-FFT estimates object velocity from samples selected in the corresponding range bin, yielding a Range-Doppler spectrum.
  • FMCW radar capture: Multiple receive antennas and Angle-FFT derive object angle and position, while beamforming can improve angular resolution.

2) Data capture based on 24/60GHz probe:

The survey describes 24/60GHz probes as using channel impulse responses rather than FMCW beat frequencies to localize reflected targets. Their captured signals support range-angle, range-Doppler, phase-waveform, and point-cloud representations, each suited to different sensing tasks.

  • Signal forms: Range-Angle spectra encode reflected signal intensity and phase by spatial position, supporting object detection and micro-displacement analysis.
  • Signal forms: Range-Doppler spectra represent moving objects and their velocities, enabling separation of objects with different velocities at the same range.
  • Signal forms: Phase waveforms characterize fine-grained human micro-displacement and are therefore suited to vital sensing.
  • Signal forms: Point clouds represent sparse reflection points on object surfaces and support tasks such as human imaging.A common generation pipeline uses Range-FFT, Moving Target Indication, and MVDR.
  • Signal preprocessing: CFAR, spectrum subtraction, fitting, filtering, and clustering address background noise, distorted phase, and dispersed point-cloud reflections.The survey links these preprocessing methods to the signal forms and sensing tasks where they are applicable.

D. Feature Extraction

The survey describes a sensing pipeline that denoises signals, extracts task-specific time- or frequency-domain features, and applies domain-knowledge or deep-learning models. Technique selection depends on sensing granularity and task requirements, with point clouds favoring contour sensing and phase/frequency analysis favoring micro-motion sensing.

  • Feature extraction: Time-domain methods analyze periodicity, duration, and amplitude, while frequency-domain methods extract spectrum, periodicity, and power-spectrum features.
  • Time-domain analysis: Template matching and signal decomposition separate or characterize vital-sign waveforms, including heart rate and breathing rate.
  • Frequency-domain analysis: Frequency-domain techniques such as Fourier, wavelet, and short-time Fourier transforms extract stable and distinguishable features while resisting noise.
  • Sensing models: Domain-knowledge models use expert-designed representations, whereas deep-learning models automatically learn features from signals such as spectrograms and point clouds.
  • Sensing models: Domain-knowledge methods have limited flexibility for nonlinear or uncertain signals, while deep learning requires substantial data and may be less interpretable.
  • Technique selection: Point clouds and deep-learning models suit contour tasks, whereas phase waveforms and frequency-domain analysis suit micro-motion tasks such as vital sensing and sound recognition.

B. Motion Recognition

mmWave motion recognition covers activities, gestures, and handwriting, using representations including CIR responses, micro-Doppler spectra, and point clouds. The surveyed systems address environmental noise, sparse measurements, and segmentation while supporting real-world recognition applications.

  • Scope: Motion recognition includes activity recognition, gesture recognition, and handwriting tracking, which are summarized as distinct application categories.
  • Activity recognition: Activity recognition requires environmental-independent features, with CIR frequency responses, micro-Doppler spectra, and voxelized point clouds used in existing systems.
  • Activity recognition: EI uses CIR frequency responses, CNN feature extraction, and classification layers to recognize activities across environments, achieving about 65% accuracy.
  • Activity recognition: SPARCS reconstructs micro-Doppler spectra from sparse communication-traffic CIR samples and reports over 0.9 F1 scores.
  • Activity recognition: m-Activity reduces point-cloud noise, accumulates 3D temporal data, and combines CNN and recurrent models for real-time activity recognition in noisy environments.
  • Gesture recognition: Gesture systems use Range-Doppler spectra, Doppler spreads, point clouds, and concentrated position-Doppler profiles for fine-grained recognition and interaction.

2) Gesture recognition:

Gesture-recognition systems address segmentation, interference, and fine-grained motion through Range-Doppler spectra, phase changes, point clouds, and specialized models. Applications include hand hygiene, sign language, handwriting, and virtual keyboards.

  • Gesture recognition: RFWash uses segmentation-free recognition because gesture segmentation is difficult and can seriously affect classification accuracy.
  • Gesture recognition: mmASL uses Doppler spreads from 60GHz signals for American Sign Language recognition, achieving 87% average sign-recognition accuracy.
  • Gesture recognition: mHomeGes combines concentrated position-Doppler profiles, a CNN, ghost-image separation, and HMM-based voting for continuous arm-gesture recognition under multipath interference.
  • Handwriting tracking: mmWave handwriting tracking targets quantitative finger or pen displacement, where background interference can distort phase changes used for micro-displacement measurement.
  • Handwriting tracking: mTrack tracks handwriting through beam steering, RSS, relative angles, and phase changes, using separate modules for anchor acquisition, phase tracking, and touch detection.
  • Gesture recognition: mmKey uses CIR amplitude differences, motion filtering, background cancellation, and MUSIC to localize keystrokes with over 95% single-key and over 90% multi-key accuracy.

C. Biometric Measurement

The survey groups biometric measurement into gait recognition, vital sensing, and sound recognition, reviewing signal forms, processing methods, and representative systems. These works support identification, continuous vital monitoring, and recovery of fine-grained physiological waveforms.

  • C. Biometric Measurement: Biometric measurement covers gait recognition, vital sensing, and sound recognition as distinct mmWave human-sensing tasks.The survey summarizes these works in Table VII and organizes them by sensing task.
  • 1) Gait recognition: 90% single-person and 88% five-person identification accuracy are reported for mmGait using point-cloud attributes and mmGaitNet.The inputs include spatial location, radial speed, and signal strength features extracted and fused by the network.
  • 1) Gait recognition: 97% single-person and over 92% four-person identification accuracy are reported for MU-ID using separated lower-limb motion features.MU-ID converts radar data into Range-Doppler spectra, separates users using angle-of-arrival and silhouette analysis, and classifies them with a CNN.
  • 2) Vital sensing: RF-SCG recovers seismocardiogram waveforms with correlation coefficients above 0.72 and times five cardiovascular events with median errors of 0.26%-1.29%.It uses a CNN-based RF-to-SCG translator and an adapted U-Net to label cardiovascular fiducial points.
  • 2) Vital sensing: MoVi-Fi addresses nonlinear mixing between movement and vital reflections with deep contrastive learning, while VED reports heart-rate median error below 2.4% and waveform cosine similarity above 0.92.These methods recover heartbeat and breathing waveforms under movement or estimate heartbeat signals from raw data.
  • 2) Vital sensing: CardiacWave extracts cardiac electromagnetic-field modulation from IF signals to recover ECG-like waveforms containing high-fidelity heart characteristics.Its CaSE mechanism differs from approaches based on chest motion caused by the heartbeat.

3) Sound recognition:

mmWave sound recognition uses reflected signals to sense speech, authenticate speakers, or eavesdrop when microphones are degraded by noise or soundproof environments. The main challenges are frequency-domain noise and accurately locating the speaker’s throat, especially in changing or NLoS settings.

  • 3) Sound recognition: mmWave sound recognition targets speech sensing in noisy environments by exploiting the penetrability and directivity of reflected signals.The survey contrasts this with microphone-based approaches that can fail outside soundproof scenes or under environmental noise.
  • 3) Sound recognition: Frequency-domain noise and inaccurate throat localization are the central challenges because speech reconstruction depends on tiny, voice-related near-throat signals.Noise may arise from acoustic fields, electromagnetic interference, or imperfect hardware.
  • 3) Sound recognition: WaveEar uses a customized 24GHz probe with 16-antenna Tx and Rx arrays, leveraging strong correlation between skin-reflected mmWave signals and speech.The probe follows a 4 × 4 antenna layout to measure vocal-cord-related throat vibration.
  • 3) Sound recognition: VocalPrint achieves over 96% authentication accuracy under different conditions by extracting text-independent vocal biometric features from the near-throat region.It uses RPCC and MFCC features with a classifier and runs on a TI AWR1642BOOST radar.
  • 3) Sound recognition: Wavoice combines mmWave and audio signals to detect voice activity and fuse multimodal features, while AmbiEar uses vibrations of surrounding objects for NLoS voice sensing.These approaches address speaker-position changes and missing direct throat-to-radar paths.

D. Human Imaging

Human imaging with mmWave supports pose, face, shape, and depth sensing, including through clothing and under low visibility. The survey emphasizes sparse spatial sampling, limited aperture, SAR motion requirements, and learning-based reconstruction as central design issues.

  • D. Human Imaging: mmWave imaging reconstructs 2D or 3D human images from reflected signals and can operate through clothing and in low-visibility conditions.Its millimeter-scale ranging resolution and privacy properties motivate applications such as pose and posture tracking.
  • D. Human Imaging: 28 cm resolution at 1 m is reported for a 1.8 cm × 1.8 cm antenna array, illustrating the data-sparsity and aperture limitations of mmWave imaging.SAR improves resolution by collecting reflections along a predetermined, densely sampled trajectory.
  • D. Human Imaging: mmFace emulates a large aperture by moving a COTS radar along a 2D slide rail and reports 96% average authentication success rate with EER below 5%.The system extracts facial biometric features from reflected-signal amplitudes and stores theoretical-model-based facial templates.
  • D. Human Imaging: MILLIPOINT enables SAR imaging on low-cost commodity radar despite uncertain vehicle motion by using cross-movement correlation and dynamic-programming self-tracking.The approach targets 3D point-cloud generation when autonomous-driving trajectories make precise localization and uniform sampling difficult.
  • D. Human Imaging: MilliPose predicts dynamically changing skeleton poses with a GRU-based recurrent model and structured prediction layer, reporting 2.1 cm median joint-location error.The predicted pose is fed back to a conditional GAN to generate a high-quality body shape.
  • D. Human Imaging: mmEye uses joint transmitter smoothing, background cancellation, and MMSE estimation on a single 60GHz WiFi device, achieving 27.2% median silhouette difference and 7.6 cm median boundary-keypoint precision.The method addresses MUSIC rank deficiency and transforms detected points of interest into plain images.

E. Lessons Learned

The survey recommends matching hardware and signal representations to sensing tasks, then tailoring algorithms and deployment to task-specific challenges. It also emphasizes varied real-world evaluation to assess performance beyond controlled scenarios.

  • E. Lessons Learned: Point clouds suit contour-related tasks such as activity recognition and human imaging, whereas phase waveforms suit fine-grained quantitative tasks such as handwriting tracking and biometric measurement.Signal form selection is presented as an initial design decision for a sensing system.
  • E. Lessons Learned: Tracking remains challenged by trajectory crossing and multipath, while continuous motion recognition and changing human orientation remain open problems.These limitations particularly affect indoor multi-person localization and dynamic motion understanding.
  • E. Lessons Learned: Biometric measurement requires precise localization of body parts and enhancement of weak signals because human position and posture are uncertain.The relevant body regions include the chest and throat.
  • E. Lessons Learned: Human imaging still requires improvements in imaging speed and model generalization because SAR and deep-learning models are widely used.The survey identifies these as further research directions for imaging systems.
  • E. Lessons Learned: Human orientation and radar deployment should be considered carefully to reduce signal distortion and environmental noise.Reflections involving people and environmental objects may also provide additional human-related information.
  • E. Lessons Learned: Multi-scenario experimental verification can reveal capabilities and support deployment in complex settings such as smart cockpits and smart homes.The survey recommends evaluating systems across various scenarios rather than relying on a single deployment condition.

VI. APPLICATION SCENARIOS

mmWave-based human sensing supports smart-home, smart-health, smart-vehicle, and security applications through contactless, fine-grained sensing. These applications include interaction, monitoring, assistance, authentication, and privacy-sensitive sensing.

  • Smart home: Smart-home systems use activity, gesture, sound, tracking, and localization sensing for device interaction, personalized services, safety monitoring, and intrusion detection.Examples include gesture-controlled devices, location-aware television control, and intruder identification through gait or voiceprint.
  • Smart home: 15 mmWave sensing is expected to provide more accurate intrusion detection than WiFi-based systems because of its high spatial resolution and fine-grained sensing.
  • Smart health: Smart-health applications provide continuous, non-intrusive monitoring of vital signs and activities, including heart rate, breathing rate, blood-pressure changes, falls, bumps, and chokes.Contactless sensing can support chronic-disease monitoring, anomaly detection, and assistance for vulnerable people.
  • Smart vehicle: In smart vehicles, mmWave sensing can identify pedestrians and cyclists in fog or rain through human imaging or activity recognition, supporting assisted driving.
  • Security: Security applications include identity authentication through gait, voiceprint, and facial features, but mmWave sensing can also expose private voices and entered passwords.The same fine-grained sensing capability supports authentication and creates privacy risks.

A. Hardware and Platforms

Current mmWave platforms are constrained by antenna count, power, denoising capability, sensing range, deployment complexity, and limited ground-truth data. The survey points toward smaller, higher-antenna, customizable hardware and more quantitative, ubiquitous applications.

  • Current limitations: Current mmWave hardware is limited by insufficient antennas, transmission power, and hardware denoising capability.
  • Current limitations: 15° angle resolution from 3 TX and 4 RX antennas is insufficient for fine-grained tasks such as face imaging.Increasing angular resolution can require beamforming, synthetic aperture methods, or larger arrays with increased size and power consumption.
  • Current limitations: Human sensing range remains short because high-frequency mmWave signals attenuate rapidly; reported examples include 5 m for mmTrack and 2 m for WaveEar.Higher transmission power or auxiliary reflectors can extend range.
  • Current limitations: Hardware imperfections and environmental interference require explicit noise modeling and reduction, as illustrated by mmSpy’s correction of oscillator ramp/settle-time noise.
  • Promising directions: Future hardware is expected to be miniaturized, contain more antennas, and support flexible customization, while applications become more quantitative and ubiquitous.The proposed direction addresses deployment, angular resolution, sensing range, scenario coverage, and richer sensing capabilities.
  • Promising directions: NLoS blockage, multipath, and missing quantitative datasets remain deployment challenges, while future applications should measure human quantities rather than only classify categories.The survey identifies quantitative sensing, wider coverage, and more universal scenarios as open directions.
  • Promising directions: Ubiquitous sensing remains open because range, occlusion, and multipath limit most existing systems to single tasks in specific scenarios.
  • Promising directions: Directional and fine-grained mmWave sensing could support applications such as skin-disease detection and blink recognition.

C. Novel Sensing Schemes

The survey discusses fusion sensing, multitask sensing, sensing side channels, and communication-oriented wireless media as ways to expand mmWave sensing. These approaches improve coverage, robustness, capability, or deployment efficiency while introducing new coupling and interference challenges.

  • Fusion sensing: Fusion sensing combines mmWave with vision, IMU, or acoustic sensing to divide tasks or mutually correct measurements for more accurate and robust results.Examples include mmWave-camera tracking, IMU-assisted SLAM, and microphone-assisted speech recognition.
  • Multitask sensing: Multitask sensing could simplify deployment by sensing gesture and vital signals with one radar, but their coupled reflections make separation and analysis difficult.The same task relationships may also enable mutual improvement through multitask learning.
  • Sensing side channels: Sensing side channels infer human information from effects on surrounding objects, including reflected signals from smartphone earpieces and changes in piezoelectric films.The survey identifies keyboard sensing and gait recognition as further possible side-channel tasks.
  • New wireless mediums: Backscatter, intelligent reflecting surfaces, and THz sensing are identified as wireless-medium innovations that can extend range, coverage, or sensing resolution.
  • Integration sensing and communication: ISAC aims to combine sensing and communication using mmWave networks, but allocating more power to information delivery reduces sensing capability and vice versa.Coverage and interference are additional implementation challenges, motivating auxiliary devices and interference-protection measures.

2) Sensing over backscatter:

Backscatter and intelligent reflecting surfaces address mmWave sensing range and coverage constraints, while THz sensing offers finer resolution but faces severe attenuation and low-SNR challenges. The survey situates these technologies within broader future directions for human sensing.

  • Sensing over backscatter: Backscatter tags can extend mmWave localization range by retroreflecting carrier signals toward their direction of arrival.Millimetro reports centimeter-level localization accuracy over extended distances with a customized Van Atta tag.
  • Sensing over backscatter: Configurable Van Atta arrays can create radar-readable road signs and support additional mmWave sensing capabilities.
  • Sensing with intelligent reflecting surface: Intelligent reflecting surfaces reconfigure wireless propagation through passive reflecting elements with adjustable phase shifts and amplitude variations.Their directional control is intended to improve mmWave coverage and energy efficiency.
  • Sensing with intelligent reflecting surface: IRS can redirect and reshape mmWave signals toward inaccessible positions, expanding coverage to blind spots.MilliMirror is presented as a passive metasurface prototype for this purpose.
  • Sensing with intelligent reflecting surface: IRS also creates privacy risks because reconfigurable surfaces can support concealed directional eavesdropping.
  • THz sensing: THz sensing offers higher spatial resolution for fine-grained tasks such as medical imaging, fingerprint detection, and skin-texture detection.
  • THz sensing: THz sensing faces extremely high attenuation, limited coverage, low-SNR signals, and unresolved requirements for reliable phase extraction and low-noise electronics.
  • Conclusion: The survey reviews hardware, techniques, and four sensing-task categories, then discusses challenges and future directions including new platforms and sensing media.
Loading 2308.03149v1…