Source-linked AI summary
MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing
Jianfei Yang, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, Lihua Xie
TL;DR
Existing camera- and wearable-based human sensing can be privacy intrusive or inconvenient, motivating non-intrusive multimodal wireless datasets. MM-Fi constructs a five-modality dataset with synchronized frames, annotations, and 27 action categories, and benchmarks modality and fusion performance for human pose estimation. The reported results show modality-specific strengths and improved performance from selected modality fusion, while dense-pose annotations remain limited by validation quality.
Problem
Camera-based sensing raises privacy and lighting concerns, while wearable inertial sensors require continuous user compliance.
Method
MM-Fi collects synchronized RGB, depth, LiDAR, mmWave radar, and WiFi CSI data with pose, action, position, and dense-pose annotations from human subjects.
Results
Selected modality fusion improves 3D human pose estimation, with R+L+W reaching 42.7, 72.0, and 61.0mm PA-MPJPE for S1, S2, and S3 on all activities.
Takeaways & Limitations
MM-Fi provides a benchmark for comparing sensing modalities and studying multimodal human pose estimation, action recognition, and cross-modal sensing.
Takeaways & Limitations
Dense-pose annotations are algorithm-generated and not validated, while current sensing tasks are restricted to activity- and keypoint-level resolutions.
Abstract
from arXiv · showhide
4D human perception plays an essential role in a myriad of applications, such as home automation and metaverse avatar simulation. However, existing solutions which mainly rely on cameras and wearable devices are either privacy intrusive or inconvenient to use. To address these issues, wireless sensing has emerged as a promising alternative, leveraging LiDAR, mmWave radar, and WiFi signals for device-free human sensing. In this paper, we propose MM-Fi, the first multi-modal non-intrusive 4D human dataset with 27 daily or rehabilitation action categories, to bridge the gap between wireless sensing and high-level human perception tasks. MM-Fi consists of over 320k synchronized frames of five modalities from 40 human subjects. Various annotations are provided to support potential sensing tasks, e.g., human pose estimation and action recognition. Extensive experiments have been conducted to compare the sensing capacity of each or several modalities in terms of multiple tasks. We envision that MM-Fi can contribute to wireless sensing research with respect to action recognition, human pose estimation, multi-modal learning, cross-modal supervision, and interdisciplinary healthcare research.
1 Introduction
MM-Fi addresses privacy and compliance limitations of camera- and wearable-based human sensing with a synchronized, multimodal dataset. It supports broad sensing research through diverse modalities, annotations, actions, and benchmarks.
- Dataset Overview: MM-Fi contains over 320k synchronized frames across five sensing modalities and provides annotations for human sensing tasks.The modalities are RGB, depth, LiDAR, mmWave radar, and WiFi CSI.
- Dataset Overview: MM-Fi uses a mobile ROS-based platform to capture and synchronize multiple sensors across diverse environments.
- Dataset Scope: The dataset includes 27 action categories spanning 14 daily activities and 13 clinically suggested rehabilitation actions.
- Research Opportunities: Its modality and annotation diversity enables multimodal fusion, cross-modal supervision, and previously unexplored multimodal pose-estimation and domain-generalization tasks.
- Research Opportunities: The authors release preprocessed data, annotations, data-loading code, and benchmarks for multimodal pose estimation and skeleton-based action recognition.
2 Related Work
Prior human-pose datasets primarily use RGB or depth, while newer datasets add individual radio-frequency or LiDAR modalities. MM-Fi targets the remaining need for datasets combining more non-intrusive modalities with varied annotations.
- Human Pose Estimation: Marker-less sensing is widely accepted for 3D human pose estimation because it is non-intrusive and user-friendly.
- Multi-modal Human Dataset: MM-Fi combines five non-intrusive sensors with action, 3D position, 2D and 3D keypoint, and 3D dense-pose annotations.
- Human Pose Estimation: Existing datasets commonly rely on RGB or depth, while later datasets explore mmWave, LiDAR, or WiFi sensing for pose estimation.
- Multi-modal Human Dataset: Earlier datasets support no more than three non-intrusive modalities, limiting development of multimodal sensing systems.
3 Sensor Platform
The MM-Fi platform integrates cameras, LiDAR, mmWave radar, and WiFi access points on a mobile ROS-enabled system. Its design supports synchronized multimodal acquisition while exposing modality-specific sensing characteristics.
- Platform Design: The customized platform integrates an Intel RealSense D435 camera, TI IWR6843 mmWave radar, Ouster OS1 32-line LiDAR, two TP-Link N750 WiFi APs, and a mini-PC.
- Platform Design: During collection, subjects stand 3.0m from the platform, while the WiFi transmitter is positioned 0.75m from the subject.
- Sensor Modalities: The RealSense system captures RGB-D frames using one RGB camera and two infrared cameras, with resolutions of 1920×1080 and 1280×720.
- Sensor Modalities: LiDAR provides dense point clouds, whereas mmWave radar produces sparse points containing spatial coordinates, Doppler velocity, and signal intensity.
- Sensor Modalities: The WiFi system uses two TP-Link N750 access points and applies a sliding-window average to produce a CSI stream of about 100Hz.
- Synchronization: ROS timestamps all modalities and retrieves frames nearest a 10Hz sampling timestamp, keeping synchronization error within 25ms.
4 Dataset
MM-Fi combines multimodal sensing, diverse action categories, and extensive annotations for human sensing research. Its processing pipeline produces synchronized data and quality-checked pose, position, dense-pose, and temporal-action labels.
- Categories of Human Motions: MM-Fi includes 27 human-action categories spanning daily activities and rehabilitation exercises for smart-home and healthcare applications.The categories comprise 14 daily activities and 13 rehabilitation actions.
- Data Annotation and Processing: 2D and 3D pose annotations are constructed from HRNetw48 detections, two-view triangulation, and optimization using reprojection, smoothness, bone-length, and action constraints.The optimized 3D keypoints are treated as ground truth for 3D keypoints.
- Data Annotation and Processing: The dataset additionally provides 3D subject-position cubes, RGB-derived 3D dense-pose labels, and temporal action-segment labels for non-RGB sensing research.The average error of the annotated 3D position cube is within 50 mm on 2,000 randomly sampled frames.
- Data Annotation and Processing: MM-Fi data are sampled at 10Hz with a uniform coordinate system and supplied in formats supported by a PyTorch loader for one or multiple modalities.The loader is designed to load multimodal data conveniently.
- Data Annotation and Processing: Re-projection quality reaches 95.66% PCKh@0.5 across 2,700 manually annotated frames, supporting the quality of the 3D keypoint annotations.The evaluation manually annotates 100 frames for each of the 27 action categories.
- Intended Uses: MM-Fi supports multimodal pose estimation, self-supervised sensing, cross-domain HPE, and few-shot action recognition through diverse modalities, subjects, environments, and annotations.These intended uses include both existing and novel human-sensing tasks.
5 Benchmark and Evaluation
The benchmark evaluates single- and multi-modal 3D human pose estimation across activity protocols and random, cross-subject, and cross-environment splits. LiDAR and mmWave radar are strongest in complementary metrics, while fusion improves robustness and accuracy.
- Benchmark Setup: Three protocols cover 14 daily activities, 13 rehabilitation exercises, and all 27 activities.The protocols distinguish free-space activities from fixed-location rehabilitation exercises.
- Benchmark Setup: Three splits test random generalization, subject differences, and environment changes.S1 uses a 3:1 sample split, S2 uses 32 training and 8 testing subjects, and S3 uses 3 training and 1 testing environment.
- Benchmark Setup: MPJPE measures pelvis-aligned Euclidean joint error, whereas PA-MPJPE additionally applies similarity alignment and focuses on joint quality.Both metrics are reported as means and standard deviations across 3 runs.
- Results and Analytics: 98.1, 94.9, and 92.5mm MPJPE make LiDAR the most stable random-split modality across P1, P2, and P3.mmWave achieves the best random-split PA-MPJPE values: 55.6, 55.3, and 57.3mm for P1, P2, and P3.
- Results and Analytics: LiDAR and mmWave radar generalize well across subjects, with their PA-MPJPE changing within 3mm from S1 across all protocols.WiFi performance declines significantly under cross-subject evaluation because limited resolution captures subtle motions and subject differences poorly.
- Results and Analytics: 166.2, 168.0, and 161.6mm MPJPE make mmWave radar the strongest cross-environment modality for P1, P2, and P3.All modalities deteriorate in S3; environmental changes particularly affect LiDAR and WiFi through nearby-object points and multipath effects.
- Results and Analytics: 42.7, 72.0, and 61.0mm PA-MPJPE are achieved by R+L+W for P3 under S1, S2, and S3, respectively.RGB-plus-LiDAR also significantly outperforms LiDAR alone for MPJPE under all three settings.
6 Limitations of MM-Fi
MM-Fi has limitations in annotation quality, benchmark maturity, and collection conditions that constrain some current uses and evaluations.
- Limitations: Manual annotations have limited quality, while dense-pose labels are algorithmically estimated and not validated.The authors plan motion-capture annotation for dense pose in MM-Fi V2.0.
- Limitations: Some multi-modal tasks remain insufficiently studied, so the provided baselines may not perform optimally without careful design.The authors present these baselines to encourage exploration and future benchmarking.
- Limitations: The current dataset is collected under controlled conditions, including a 3m distance and the same facing direction.This setting limits the breadth of the collection conditions.
7 Conclusion
The paper presents MM-Fi as a comprehensive non-intrusive multi-modal 4D human pose dataset and benchmark for wireless human sensing. Its synchronized data and multi-modal evaluations support research across sensing, computing, vision, and healthcare.
- Conclusion: MM-Fi contains 320.76k synchronized frames, 27 pose categories, and data from 40 subjects.The conclusion describes RGB-D, LiDAR, mmWave radar, and WiFi as the four sensing modalities.
- Conclusion: The dataset and benchmarks support multi-modal human pose estimation and help researchers select sensor modalities according to their advantages and disadvantages.The paper positions MM-Fi for research in machine learning, ubiquitous computing, computer vision, mobile computing, and healthcare.
A.1 Ethics Statement
The study received institutional ethics approval and used de-identified, voluntarily provided data for public research release.
- Ethics Statement: The Nanyang Technological University IRB committee reviewed and approved the human subject study under IRB-2022-1067.Subjects were informed that de-identified data would be publicly available for research.
- Ethics Statement: Facial blur was used to de-identify the MM-Fi data.The recruitment was voluntary, and the paper states that the research does not directly endanger any person.
A.2 Dataset Documentation
The dataset documentation describes MM-Fi’s action coverage, multimodal data organization, diverse environments, processing pipeline, and annotation procedure.
- Dataset composition: MM-Fi includes daily and rehabilitation actions, with 40 volunteers distributed across four environmental settings.Ten volunteers participated in each environment, and the action set is summarized in Table 5.
- Distribution and maintenance: The dataset is maintained through a GitHub repository with a registered DOI and multiple cloud-storage mirrors.The repository includes recommended download and usage instructions, while Google Drive, Baidu Netdisk, and Alibaba Cloud support accessibility and preservation.
- Data processing and formats: Data from different modalities are converted into common structured formats, while LiDAR point clouds are filtered to subject-centered regions of interest.The toolbox provides PyTorch data loading, and the processing pipeline addresses modality-specific input structures.
- Environmental settings: Four environmental settings are created by deploying the sensor platform horizontally and vertically across two rooms with shared dimensions.Room 1 and Room 2 measure 8.5 m by 7.8 m, while platform orientation contributes to environmental diversity.
- Temporal annotation: Temporal segments are annotated by human annotators using an automated frame-by-frame program and manually selected breakpoints.The process supports temporal action segmentation and finer-grained samples.
- Pose annotation: Occlusion-aware annotation adds regularization based on joint relationships and sensing-area constraints.For example, the loss regularizes unobservable hip and shoulder joints and constrains poses from bending backward under the observed coordinates.
A.3 License Agreement
The work is licensed under CC BY-NC 4.0, with the authors accepting responsibility for rights violations, novelty, and privacy leaks.
- License: The paper is released under the CC BY-NC 4.0 license.The authors state that they bear responsibility for rights violations, work novelty, and privacy leaks.
A.4 Benchmark Implementation Details
The benchmarks implement modality-specific baselines and evaluate human pose estimation and skeleton-based action recognition across MM-Fi’s sensing modalities. Results show strong RGB, LiDAR, and mmWave pose visualizations, while WiFi is limited for pose and action recognition.
- Implementation: Baseline implementations match original-paper hyperparameters, use PyTorch, and report means and standard deviations over three random seeds.The benchmark code and modality-specific manuals are provided through the linked repository.
- Human pose estimation: VideoPose3D serves as the RGB human-pose baseline, while Point Transformer is used for LiDAR and mmWave point-cloud inputs.MetaFi++ is used as the WiFi baseline, incorporating convolution and self-attention for antenna-information fusion.
- Human pose estimation: RGB, LiDAR, and mmWave radar produce relatively accurate human-pose visualizations, whereas WiFi-based pose estimation is unsatisfactory because of resolution limitations.These comparisons are illustrated in Figure 5.
- Action recognition: Skeleton-based action recognition uses 3D joints from human-pose estimation, divides sequences into 30-frame clips, and evaluates subject-level training and testing splits.The recognition benchmark uses seven subjects for training and three for testing.
- Action recognition: RGB yields the best action-recognition results for AGCN and CTRGCN, mmWave ranks second, and WiFi cannot enable action recognition because predicted joints are not robustly consistent.The authors attribute mmWave’s advantage over LiDAR to smoother sequence predictions and report that AGCN outperforms CTRGCN on MM-Fi modalities.
- Research directions: MM-Fi supports cross-domain wireless-sensing studies, cross-modal supervision for fine-grained tasks, and multimodal fusion across five sensing modalities.The proposed research directions include domain adaptation, pose estimation, action segmentation, and complementary sensing such as RGB plus WiFi.