Person-in-WiFi: Fine-grained Person Perception using WiFi

Fei Wang; Sanping Zhou; Stanislav Panev; Jinsong Han; Dong Huang

Source-linked AI summary

Person-in-WiFi: Fine-grained Person Perception using WiFi

Fei Wang, Sanping Zhou, Stanislav Panev, Jinsong Han, Dong Huang

arXiv:1904.00276v1Mar 30, 2019cs.CV

TL;DR

Fine-grained body segmentation and pose estimation traditionally rely on spatially resolved camera, radar, or LiDAR data, whereas WiFi provides only one-dimensional signal summaries. Person-in-WiFi reconstructs 2D body structure from WiFi using video annotations and deep learning, achieving person perception comparable to 2D-image approaches.

Problem
Fine-grained person perception has been little explored with WiFi because each antenna receives a one-dimensional signal summary instead of spatially resolved body measurements.
Method
The approach uses off-the-shelf transmitter and receiver WiFi antennas, RGB-video annotations, and deep learning to reconstruct 2D segmentation masks and joint coordinates from WiFi signals.
Results
Person-in-WiFi achieved person perception comparable to approaches using 2D images across body segmentation and pose estimation experiments.
Takeaways & Limitations
Fine-grained person perception can be achieved with pervasive WiFi antennas, including reconstruction of 2D human-body information from 1D data.
Takeaways & Limitations
Current results remain constrained by WiFi spatial resolution, rare poses, and incomplete camera-based annotations.

Abstract

from arXiv · show

Fine-grained person perception such as body segmentation and pose estimation has been achieved with many 2D and 3D sensors such as RGB/depth cameras, radars (e.g., RF-Pose) and LiDARs. These sensors capture 2D pixels or 3D point clouds of person bodies with high spatial resolution, such that the existing Convolutional Neural Networks can be directly applied for perception. In this paper, we take one step forward to show that fine-grained person perception is possible even with 1D sensors: WiFi antennas. To our knowledge, this is the first work to perceive persons with pervasive WiFi devices, which is cheaper and power efficient than radars and LiDARs, invariant to illumination, and has little privacy concern comparing to cameras. We used two sets of off-the-shelf WiFi antennas to acquire signals, i.e., one transmitter set and one receiver set. Each set contains three antennas lined-up as a regular household WiFi router. The WiFi signal generated by a transmitter antenna, penetrates through and reflects on human bodies, furniture and walls, and then superposes at a receiver antenna as a 1D signal sample (instead of 2D pixels or 3D point clouds). We developed a deep learning approach that uses annotations on 2D images, takes the received 1D WiFi signals as inputs, and performs body segmentation and pose estimation in an end-to-end manner. Experimental results on over 100000 frames under 16 indoor scenes demonstrate that Person-in-WiFi achieved person perception comparable to approaches using 2D images.

1. Introduction

The paper addresses the largely unexplored problem of fine-grained person perception from one-dimensional WiFi signals. It proposes mapping WiFi measurements to 2D body structure using deep learning and video-derived annotations.

Motivation: Cameras, radars, and LiDARs support fine-grained perception but require spatially resolved outputs and face lighting, occlusion, privacy, cost, or power constraints.The cited sensors produce images, depth maps, or point clouds with substantial spatial resolution.
Challenge: WiFi antennas provide only a one-dimensional summary of 3D space, making fine-grained reconstruction severely ill-posed.Human bodies and environments jointly affect signals through multipath, while body differences and temporal changes add difficulty.
Approach: The system uses transmitter and receiver antenna sets, video-derived body annotations, and deep learning to map WiFi signals to segmentation masks and joint coordinates.The antenna arrangement supplies multiple signal patterns, while RGB videos provide segmentation and pose annotations.
Contribution: Fine-grained person perception can be achieved using pervasive WiFi antennas.The paper identifies this as a first demonstration of the capability.
Contribution: The approach claims that 2D human-body sensing can be performed with sensors whose physical spatial resolution is as low as one dimension.This reframes fine-grained perception as reconstruction from low-dimensional measurements rather than directly captured spatial data.

2. Related Work on Person Perception

Prior person-perception systems use cameras, radar, or LiDAR, while earlier WiFi work mainly addressed coarse localization, activity recognition, or static-object sensing. Person-in-WiFi extends WiFi sensing toward fine-grained body segmentation and pose estimation.

Camera-based: Camera-based pose systems commonly detect people, crop person regions, and estimate poses independently from cropped image features.This two-stage design benefits from spatially localized image pixels.
WiFi-based: WiFi cannot crop 2D person pixels, so the proposed pose pipeline generates joint heat maps and part affinity fields directly from WiFi signals.Person-wise poses are then computed from these representations.
Radar-based: Radar systems use FMCW measurements and time-of-flight processing to generate depth maps for localization, outlines, and fine-grained pose estimation.Reported radar systems use dedicated, synchronized hardware and specialized antenna arrays.
LiDAR-based: LiDAR-based approaches capture 3D point clouds for tasks including person detection, tracking, surveillance, and robotic mapping.The cited comparison notes that LiDAR generally offers lower angular resolution than cameras.
WiFi-based: Earlier WiFi research focused on coarse localization, closed-set activity classification, or static-object sensing rather than fine-grained person perception.Examples include activity recognition, bubble-like localization heatmaps, and antenna-swept holograms.

3. Person Perception with WiFi Signals

The WiFi sensing system reconstructs fine-grained body information from signals whose spatial content collapses into one-dimensional CSI measurements. It increases signal diversity with multiple antenna pairs and frequencies, then constrains the mapping using multiple spatial body representations and annotations.

Methodology: The system uses one transmitting antenna, one receiving antenna, and one EM frequency to probe human bodies and the environment through signal propagation.Signals penetrate, refract, and reflect at many points before reaching the receiver.
Methodology: A single CSI value collapses spatial information, making reconstruction of a 100×100 body image an ill-posed problem with 10^4 unknowns.The formulation is I(P) = f(H), where one CSI signal must explain the body image.
Methodology: Three transmitting antennas, three receiving antennas, and 30 EM frequencies provide nine propagation pairs and diverse superposition patterns.Different antenna paths and wavelengths capture signals at different paths and object scales.
Methodology: The approach constrains the mapping by learning multiple spatial body representations, including body masks, joints, and limbs, with multi-task deep neural networks.These representations share the same spatial layout while emphasizing different body structures.
WiFi Signal, CSI and Hardware: CSI is computed from signals between each antenna pair at each OFDM frequency, using transmitted and received digital signals.The paper defines each CSI sample as c_i = y_i/x_i.

4. Deep Learning for Person-in-WiFi

Person-in-WiFi maps CSI inputs to complementary 2D body representations for segmentation and pose estimation. Its multi-task network uses spatially aware processing and Matthew Weight to address sparse joint signals during training.

Data and annotations: The system uses CSI collected from six scenes and synchronized RGB video annotations from activities involving one to five concurrent persons.CSI was recorded at 100 Hz, while RGB videos were recorded at 20 FPS and used only for annotation.
Data and annotations: RGB-derived annotations include a 1 × 46 × 82 segmentation tensor, 26 joint-heatmap channels, and 52 part-affinity-field channels.The joint heatmaps represent 25 joints plus background, while the affinity fields encode x and y coordinates for 26 limbs.
Network outputs: CSI is mapped to segmentation masks, joint heatmaps, and part affinity fields for person perception.Joint heatmaps and part affinity fields are later used for multi-person joint association.
Network architecture: Nine transmitter–receiver views and receptive fields spanning the upsampled input force features to match the spatial layout of the target representations.The 3 × 3 antenna-pair dimension captures multiple views, while supervision aligns feature maps with masks, joints, and limbs.
Loss and Matthew Weight: The training objective combines segmentation, joint-heatmap, and part-affinity-field losses with scalar weights.The segmentation loss is empirically downweighted relative to the joint and affinity-field losses.
Loss and Matthew Weight: Because joints occupy less than 2% of joint-heatmap pixels, direct L2 loss tends to average errors over background regions.The paper reports that 98% of pixels are background in one example joint-heatmap tensor.
Loss and Matthew Weight: Matthew Weight increases attention on larger joint elements and significantly improves pose estimation compared with direct L2 loss.The weighting is also applied when computing the part-affinity-field loss.

5. Experiments

Experiments evaluate WiFi-based body segmentation and pose estimation using automatically annotated, synchronized CSI and video data. Results show useful perception, deployment gains across environments, and failure modes tied to spatial resolution, rare poses, and annotation coverage.

Experimental setup: Data used synchronized CSI samples and monocamera annotations from Mask R-CNN and OpenPose for groups of 1–5 subjects.The experiment is described as a proof of concept that could be improved with manual annotations and multi-camera images.
Metrics: Body segmentation was evaluated with mIoU and mAP over AP@50 to AP@95, while pose estimation used modified PCK over 25 joints.PCK normalizes prediction error by the diagonal length of the person bounding box.
Body segmentation: 0.38 mAP over AP@50-AP@95 was achieved for body segmentation, with body locations, torsos, and legs generally segmented well.Higher AP@50-AP@70 values indicate detectable person profiles, while lower AP@80-AP@95 values indicate weaker subtle-mask detection.
Pose estimation: Most body joints achieved high PCK at 0.1 normalized distance error, with Torso&Arms and Legs outperforming Head and Feet.Figure 11 groups the 25 joints into Head, Torso&Arms, Legs, and Feet.
Comparison with camera-based approaches: The models are bounded by Mask R-CNN and OpenPose annotations, and manually annotated comparisons showed noticeable quantitative gaps.The authors suggest more data and higher-quality annotations could reduce these gaps.
Failure cases: Failure cases include small limbs missed through diffraction, rare poses, and incomplete single-camera annotations caused by narrower field of view.The paper suggests higher small-limb regression weights, more data, augmentation, temporal smoothing, and multi-camera annotation as remedies.
Deployment in untrained environments: In 14 training scenes and 2 testing scenes, adversarial training improved segmentation mIoU from 0.12 to 0.24 and pose mPCK@0.20 from 19.34 to 31.06.Further improvement in untrained environments requires more data and annotations.

6. Conclusion

The conclusion presents WiFi antennas as practical sensors for fine-grained person perception from one-dimensional signals. Person-in-WiFi reconstructs two-dimensional human-body information using off-the-shelf router-like devices suitable for indoor settings.

Conclusion: WiFi devices are invariant to illumination and privacy-friendly compared with cameras, while being cheaper, smaller, and more power efficient than radars and LiDARs.The stated indoor applications include warehouses, hospitals, offices, and homes.
Conclusion: Person-in-WiFi reconstructs 2D fine-grained human-body information from 1D data received by off-the-shelf WiFi antennas.The antennas are lined up like regular household WiFi routers.