Making the Invisible Visible: Action Recognition Through Walls and Occlusions

1. Introduction

• Camera-based methods require visible subjects, and prior RF systems covered few actions, generalized poorly to new people or environments, and could not handle multi-person actions.Changing viewpoint or interpolating frames does not address fixed-camera settings or long periods of full occlusion.Source: §1 · p. 1
• RF-Action converts wireless signals into 3D skeletons, a shared representation that supports training with RF and vision data and additional skeleton supervision.The paper also presents skeletons as minimally affected by environments and subject identities, supporting generalization.Source: §1 · p. 2
• A spatio-temporal attention module uses joint confidence over time, while a multi-proposal module supports simultaneous actions by different people.The modules address errors in predicted skeleton joints and scenes where people perform different actions at once.Source: §1 · p. 2
• 87.8 mAP without occlusions and 83.0 mAP in through-wall scenarios are reported for RF-Action, which performs comparably to vision-based systems in visible settings.The evaluation dataset spans 25 hours and includes 30 individuals performing single-person and multi-person actions.Source: §1 · p. 2
• The paper presents RF-Action as a skeleton-based RF action recognizer that handles actions and interactions through walls and in extremely poor lighting.Its contributions also include multimodal knowledge transfer, spatio-temporal attention, and multi-proposal action detection.Source: §1 · pp. 1–2

2. Related Works

• Video methods progress from hand-crafted temporal descriptors to deep-learning approaches using 3D convolutions or two-stream networks.The cited approaches extract motion and appearance jointly or model spatial and temporal features separately.Source: §2 · p. 2
• Skeleton action recognition has used recurrent, convolutional, and graph neural networks, with skeletons offering concise representations and robustness to background noise.RF-Action expands the HCN model with spatio-temporal attention and a multi-proposal module.Source: §2 · pp. 2–3
• Earlier radio-based systems either recognize a maximum of 10 simple actions for one person or require sensor networks or body-worn instrumentation.The paper identifies instrumentation as limiting those systems’ utility and robustness.Source: §2 · p. 3

3. Radio Frequency Signals Primer

• The radio produces horizontal and vertical heatmaps at 30 pairs per second, with stronger values indicating stronger reflections from a location.The horizontal heatmap projects onto a plane parallel to the ground; the vertical heatmap projects onto a perpendicular plane.Source: §3 · p. 3
• RF’s depth and angle resolutions are 10 cm and 10 degrees, making similar activities difficult to distinguish.The paper gives hand waving and hair brushing as examples.Source: §3 · p. 3
• Human bodies act as specular reflectors at the radio’s roughly 5 cm wavelength, so limbs may reflect signals away from the sensor or return weak reflections.Small surfaces such as hands are harder to track because they reflect less signal.Source: §3 · p. 3
• Walls attenuate RF signals more than air, weakening body reflections and reducing action-detection accuracy in wall and occlusion scenarios.Source: §3 · p. 3

4. Method

• RF-Action maps wireless signals to per-person 3D skeletons, then recognizes actions and interactions over time; visual 3D skeletons can enter the same action framework.This architecture supports training with RF-generated skeletons and existing skeleton-based action-recognition datasets.Source: §4 · p. 3, Figure 3 · p. 4

4.1. Skeleton-Generation from Wireless Signals

• A skeleton-generation network processes 3-second (90-frame) horizontal and vertical RF heatmaps through spatio-temporal convolutions, region proposals, and 3D pose estimation.The resulting proposals are used to extract multi-person 3D skeletons.Source: §4.1 · p. 4

4.2. Modality-Independent Action Recognition

• Per-person skeleton sequences encode each keypoint’s 3D coordinates and confidence across time; extracted features feed single-person and interaction proposal networks, followed by action classification.The classifier first refines each temporal proposal as action or non-action, then predicts its action class.Source: §4.2 · p. 4

4.2.1 Spatio-Temporal Attention Module

• Joint-confidence masks combine spatial and temporal latent features, while multi-head attention weights useful timestamps; experiments report improvements on RF-predicted skeletons and visual action-recognition benchmarks.The attention design changes HCN’s late fusion of spatial and temporal paths.Source: §4.2.1 · pp. 4–5

4.2.2 Multi-Proposal Module

• Because people may act or interact simultaneously, the multi-proposal module generates possible single-person actions and pairwise interactions, allowing multiple outputs with interactions prioritized.The approach replaces max-pooling across people’s features with proposals for individual actions and interactions.Source: §4.2.2 · p. 5

4.3. Multimodal End-to-end Training

• A differentiable regressor replaces argmax so action labels can supervise skeleton prediction; RF data trains both stages, while skeleton datasets train action recognition, improving performance.Gradients from RF data tune the skeleton-prediction and action-recognition models, whereas gradients from skeleton-based datasets tune the action-recognition module.Source: §4.3 · p. 5

5.1. Dataset

• RF-MMD contains 25 hours from 30 volunteers across 10 environments, with 35 actions and separate 20-hour training and 5-hour test sets.It includes two through-wall scenarios, one for training and one for testing; PKU-MMD supplies additional training examples.Source: §5.1 · pp. 5–6

5.2. Setup

• Evaluation uses mAP at IoU thresholds 0.1 and 0.5, with 3D skeletons and manually labeled action times and categories as ground truth.The 3D skeletons are triangulated from multi-view camera data.Source: §5.2 · pp. 6–7, §5.1 · p. 6

5.3. Qualitative Results

• Qualitative results show action and interaction detection in visible scenes, under occlusion, and in poor lighting; displayed skeletons are 2D projections of intermediate 3D skeletons.Figure 5’s top rows show visible scenes and its bottom rows show occluded or poorly lit scenes.Source: Figure 5 · p. 6, §5.3 · p. 7

5.4. Comparison of Different Models

• RF-Action outperforms HCN and Aryokee on visible and through-wall RF-MMD tests, with wireless signals as input.HCN receives RF-Action-generated RF skeletons for the comparison.Source: Table 1 · p. 7, §5.4 · p. 7

5.5. Comparison of Different Modalities

• Using camera-derived ground-truth skeletons yields only a few percent improvement in visible scenes, while RF-Action also works through walls with a few-percent accuracy decrease.The comparison uses a 10-view camera system versus one wireless device; wall traversal attenuates the signal.Source: §5.5 · p. 7, Table 2 · p. 7

5.6. Action Detection

• A test example shows the model detecting two people’s individual actions and their interactions with high accuracy.Figure 6 uses blue for ground-truth segments, red for detections, and frame number on the horizontal axis.Source: §5.6 · p. 7, Figure 6 · p. 8

5.7. Ablation Study

• Attention helps across RF-MMD, PKU-MMD, and NTU-RGB+D, while multimodal training with PKU-MMD improves RF-MMD performance.Attention is especially useful on RF-MMD, where RF-predicted skeleton joints may be inaccurate.Source: §5.7 · p. 8, Table 3 · p. 8, Table 5 · p. 8
• The multi-proposal module improves performance when people perform different actions simultaneously, and end-to-end training improves action detection while reducing RF-skeleton joint-location errors.The authors attribute the latter improvement to action-detection loss regularizing the generated 3D skeletons.Source: §5.7 · p. 8, Table 4 · p. 8, Table 6 · p. 8

• The paper presents RF-Action as the first model for skeleton-based action recognition using radio signals.Source: §6 · p. 8
• RF-Action demonstrates recognition of actions and interactions through walls and in extremely poor lighting.Source: §6 · p. 8
• The model is intended for settings where cameras are hard to use because of privacy concerns or poor visibility, including homes and smart-home systems.Source: §6 · p. 8