PaperlayerView exact revision on arXiv

Source-linked AI summary

WiFi-JEPA: Self-supervised Learning for WiFi-CSI 3D Human Pose Estimation

Doeon Kim, Jungyoon Lee, Seongsin Kim, Seong-heum Kim

arXiv:2607.11064v1cs.CV

TL;DR

WiFi pose estimation is challenged by environment shifts and costly camera-based labeling. WiFi-JEPA learns CSI representations through masked latent prediction and ray-traced simulation, and improves single- and multi-person results on PiW3D over prior WiFi baselines.

  • Problem

    WiFi pose estimation faces environment-shift fragility and limited scale from camera-based annotation pipelines, motivating CSI representations that preserve signal structure without dense camera-derived labels.

    Source: §1 · p. 2

  • Method

    WiFi-JEPA uses CSI-specific tokenization and link masking to predict masked latent representations, alongside ray-traced simulated CSI for pre-training.

    Source: §1 · p. 3

  • Results

    76.8 mm single-person and 93.5 mm multi-person MPJPE on PiW3D outperform prior WiFi HPE baselines under the same evaluation protocol.

    Source: §1 · p. 3

  • Takeaways & Limitations

    WiFi-JEPA consistently improves downstream pose estimation, while four vision-native SSL objectives degrade performance below training from scratch.

    Source: §6 · p. 15

    Geometric primitives differ substantially from human bodies in scattering behavior, so the claim that channel-dynamics diversity can suffice is framed as a hypothesis.

    Source: §3 · p. 5

Abstract

from arXiv · show

WiFi Channel State Information (CSI) enables privacy-preserving human pose sensing in camera-denied environments, but existing WiFi-based pose estimators often fail under environment shifts and rely on costly camera-based annotation pipelines that limit scale. We propose WiFi-JEPA, a self-supervised framework that learns CSI-native representations by predicting masked latent embeddings instead of reconstructing raw CSI signals that may contain hardware-specific artifacts. WiFi-JEPA makes three contributions: (i) CSI-specific tokenization and link masking tailored to the CSI tensor over channel, time, and link (C,T,L); masking entire Tx-Rx antenna links forces the model to predict one spatial link view from others, capturing cross-link correlations informative of 3D spatial structure. (ii) A ray-tracing CSI simulation pipeline that generates diverse unlabeled CSI from randomized geometric primitives, providing scalable pre-training data without pose annotations. (iii) State-of-the-art results on Person-in-WiFi-3D: WiFi-JEPA outperforms prior WiFi-CSI baselines on both single- and multi-person 3D pose estimation under the same evaluation protocol. We also show that simulated CSI provides complementary pre-training signal to real CSI, and that four vision-native SSL objectives degrade performance below training from scratch, whereas WiFi-JEPA consistently improves downstream pose estimation.

1 Introduction

WiFi-JEPA addresses fragility to environment shifts, limited labeled data, and CSI’s hardware-dependent structure with self-supervised latent prediction and CSI-specific modeling. The paper reports improved single- and multi-person pose estimation on PiW3D and introduces ray-traced simulated CSI for pre-training.

1 Introduction

  • Human pose estimation supports health monitoring, human–computer interaction, and safety-critical perception, while vision can be limited by occlusion, low illumination, and privacy constraints.These constraints motivate pose sensing in camera-denied scenarios.Source: §1 · p. 1
  • WiFi-based HPE faces fragile cross-domain generalization, limited camera-derived labels, and CSI representations that can conflate frequency, time, and link structure.These challenges motivate representations that preserve CSI structure while addressing environment and hardware shifts without dense camera-derived labels.Source: §1 · p. 2
  • WiFi-JEPA predicts masked latent representations from CSI-specific tokens, while link masking predicts hidden Tx–Rx observations from the remaining links.The framework also introduces ray-tracing CSI simulation to reduce dependence on scarce labels.Source: §1 · p. 3
  • 76.8 mm single-person and 93.5 mm multi-person MPJPE on PiW3D outperform prior WiFi HPE baselines under the same evaluation protocol.The framework combines real and simulated pre-training data.Source: §1 · p. 3

2 Related Work

Prior work covers latent and masked-prediction SSL, WiFi pose estimation, and synthetic-data pre-training, but the cited approaches leave gaps in CSI-axis-aware learning and ray-traced self-supervised sensing.

2.1 Self-supervised representation learning

  • JEPA predicts learned latent targets rather than reconstructing masked inputs, while prior CSI-SSL methods do not explicitly distinguish masking across CSI’s physical axes.The related work contrasts this with pixel-space MAE reconstruction and notes that prior CSI methods adapt image-domain schemes.Source: §2.1 · pp. 3–4

2.2 WiFi-based Human Pose Estimation

  • Supervised WiFi pose methods require visual supervision, and DT-Pose [9] is identified as the only prior SSL method for WiFi pose estimation.DT-Pose flattens CSI axes into one sequence and uses a decoder with explicit skeletal-topology constraints.Source: §2.2 · p. 4

2.3 Simulation data for representation learning

  • Prior synthetic-data work motivates structural diversity, but the cited related work identifies no ray-tracing self-supervised pre-training for WiFi sensing.The discussion also notes that simple moving scatterers can produce rich channel variations relevant to spatial dynamics.Source: §2.3 · p. 4

3 Simulated CSI from Geometric Primitives

The sim-object pipeline creates benchmark-compatible CSI from randomized geometric scenes, without human models or external motion data. Its design emphasizes varied channel dynamics and temporal variation for self-supervised pre-training.

3 Simulated CSI from Geometric Primitives

  • sim-object generates benchmark-compatible CSI from simple geometric shapes without human models or external motion data.The pipeline is proposed as an alternative to costly collection requiring dedicated environments, synchronized receivers, and motion-capture labels.Source: §3 · p. 5
  • The simulation hypothesis is that diverse spatio-temporal channel variation may matter more for pre-training than reproducing body-specific multipath faithfully.Randomization covers room geometry, object count, trajectory, and wall materials, although spheres and human bodies differ substantially in scattering behavior.Source: §3 · p. 5
  • Stage 1 uses a CPU-only generator to create randomized indoor scenes with geometric primitives following physics-based trajectories.Scenes vary in room dimensions, wall materials, primitive type and count, and randomized transmitter and receiver placement.Source: §3 · pp. 5–6, Figure 2 · p. 5
  • Stage 2 runs Sionna RT with 20 independent passes per frame to produce temporal variation for the CSI input.The passes sample object positions within each measurement window; the resulting tensor is shaped (60, 20, 9) for the tokenizer.Source: Figure 2 · p. 5, §3 · p. 6

4 WiFi-JEPA

WiFi-JEPA preserves CSI’s channel, time, and link structure, then uses link masking to predict latent representations from visible links. Its pre-trained encoder is fine-tuned with a PETR decoder for multi-person 3D pose estimation.

4.1 CSI-specific Tokenization

  • CSI-specific tokenization reshapes the signal to (C, T, L) = (60, 20, 9), producing 180 tokens that each represent a precise spatial-temporal coordinate.The 9 links are independent spatial viewpoints; unlike flattening, this keeps temporal and link dimensions separate.Source: Figure 3 · p. 6, §4.1 · p. 7

4.2 Link Masking

  • At a 0.6 masking ratio, link masking targets 5 of 9 links and uses the remaining 4 to predict the full temporal signal of unseen links.All time steps of each selected link are masked together; ablations report that link masking outperforms time, block, and random-token masking.Source: §4.2 · pp. 7–8, Figure 4 · p. 7

4.3 Pre-training Objective

  • The context encoder processes visible tokens, while a predictor estimates target-encoder representations at masked positions using a Smooth L1 loss.The target encoder processes all tokens and is updated by EMA with a cosine momentum schedule from 0.996 to 1.0.Source: §4.3 · p. 8

4.4 Fine-tuning for 3D Pose Estimation

  • For pose estimation, the pre-trained encoder is fine-tuned end-to-end on real CSI with a PETR decoder that predicts 14 3D keypoints per detected person.The encoder uses 0.1× the base learning rate, while the decoder and regression heads use the base rate; Hungarian matching assigns queries to instances.Source: §4.4 · p. 9

5 Experiments

The experiments evaluate WiFi-JEPA on PiW3D, a WiFi-CSI benchmark with 3D pose labels and single- and multi-person test scenes. They report MPJPE as the primary metric alongside aligned pose error and keypoint accuracy.

5 Experiments

  • The experiments assess state-of-the-art performance, the effects of SSL objectives and structure-aware design, and whether simulated CSI can complement or replace real pre-training data.PiW3D is identified as the only public WiFi-CSI benchmark with multi-person 3D pose annotations.Source: §5 · p. 9

Dataset.

  • PiW3D contains 89,946 training frames and 7,824 test frames spanning single-person, two-person, and three-person scenes across an office, classroom, and corridor.The dataset includes synchronized CSI and 14-joint pose labels from seven volunteers performing eight daily actions.Source: Dataset · p. 9

Metrics.

  • MPJPE is the primary metric; the evaluation also reports PA-MPJPE, PCK at 20 and 50 mm, and per-axis absolute errors.PA-MPJPE removes global translation, rotation, and scale; PCK measures the percentage of correct keypoints within the stated threshold.Source: Metrics · p. 9

Implementation details.

WiFi-JEPA outperforms prior WiFi-CSI baselines on PiW3D in both single- and multi-person MPJPE comparisons. The section also specifies its training setup and the methods used for comparison.

Implementation details.

  • Pre-training uses about 90K real and 90K simulated frames; the 100-epoch model uses link masking that randomly drops 60% of antenna links per sample.The context and target encoders have 12 layers and 512 dimensions; the predictor has 8 layers, and the encoder is fine-tuned at 0.1× the base learning rate.Source: Implementation details · p. 10

Comparison with State-of-the-Art

  • The comparison includes WiPose, HPE-Li, MetaFi++, DT-Pose, and PiW3D; only PiW3D reports prior multi-person results.WiPose, HPE-Li, MetaFi++, and DT-Pose are compared only in single-person scenes.Source: Comparison with State-of-the-Art · p. 10
  • 76.8 mm single-person MPJPE beats DT-Pose’s 90.0 mm and PiW3D’s 91.7 mm; 93.5 mm multi-person MPJPE improves on PiW3D’s 107.2 mm.The real-only variant also surpasses all baselines, and adding simulation reduces single- and multi-person MPJPE by a further 1.4 mm and 3.6 mm, respectively.Source: Comparison with State-of-the-Art · p. 10
  • Figure 6 places PiW3D [28] in the top row and WiFi-JEPA below, with one-, two-, and three-person scenes arranged left to right.It provides a qualitative comparison across increasing scene counts.Source: Figure 6 · p. 11

Challenging Scenarios

WiFi-JEPA improves pose-estimation errors in held-out environments and across multi-person scenes, including on extremity estimates. The evaluations also compare its cross-environment performance with a few-shot supervised domain-adaptation baseline.

Challenging Scenarios

  • 324.2 mm mean MPJPE—48.2% below PiW3D’s 626.4 mm—under leave-one-environment-out evaluation.Both pre-training and fine-tuning exclude the held-out environment.Source: Table 2 · p. 11, Challenging Scenarios · p. 12
  • 171.3 mm macro mean MPJPE, down from Supervised+DANN’s 208.6 mm with 10% of target-environment labels.WiFi-JEPA reduces error by 17.9% across held-out environments in this few-shot setup.Source: Challenging Scenarios · p. 12
  • Single-, two-, and three-person MPJPE falls from 91.7, 108.1, and 125.3 mm to 76.8, 96.0, and 110.7 mm, respectively.The reductions are 16.2%, 11.2%, and 11.7%, respectively.Source: Table 3 · p. 12, Challenging Scenarios · p. 12
  • 64.7 mm elbow-and-hand error, down from PiW3D’s 160.5 mm; hand depth error falls from 92.9 to 26.4 mm.The reported elbow-and-hand improvement is 59.7%.Source: Table 4 · p. 12, Challenging Scenarios · p. 12

5.3 Effect of Simulated CSI

The section sets up a comparison of simulated-CSI pre-training data and its downstream MPJPE effects, with shared model and fine-tuning conditions.

5.3 Effect of Simulated CSI

  • The sim-human variant uses Blender-animated human meshes and motion-captured clips instead of geometric primitives.Both simulated datasets use the same number of clips and are sampled to 90K frames.Source: §5.3 · p. 13
  • The sim-object versus sim-human comparison is designed to isolate scene content under matched clip counts and 90K-frame samples.The variants contrast randomized-physics geometric primitives with animated human meshes and motion-capture sequences.Source: §5.3 · p. 13
  • Table 5 compares downstream MPJPE across pre-training data while holding the encoder, decoder, and fine-tuning protocol fixed.The table defines Δ as improvement over no pre-training.Source: §5.3 · p. 12, Table 5 · p. 13

sim-object suffices for pre-training.

WiFi-JEPA performs better than the reported SSL baselines, while simulated CSI contributes pre-training value comparable to real CSI and complements it when combined. The data comparisons also find that geometric primitives outperform human meshes.

sim-object suffices for pre-training.

  • 100.1 mm MPJPE for sim-object versus 110.3 mm for sim-human; sim-human is 7.9 mm worse than no pre-training.The authors interpret this comparison as evidence that trajectory diversity may matter more than geometric realism for transfer; sim-object is also 3.0 mm from real-only pre-training at 97.1 mm.Source: sim-object suffices for pre-training · p. 13

Complementarity of Simulated and Real Data.

  • Combining 90K simulated and 90K real frames reduces MPJPE by 8.9 mm over scratch, compared with a 5.3 mm reduction from real-only pre-training.The simulated data costs approximately 10 GPU-hours on one RTX 4090.Source: Complementarity of Simulated and Real… · p. 13

Comparison with SSL Baselines

  • 33.2 mm lower MPJPE than MAE (130.3 → 97.1) and +11.7 pp in PCK@20 (47.6% → 59.3%); all four SSL baselines degrade performance relative to no pre-training (102.4 mm).PA-MPJPE remains comparable to the no-pre-training baseline.Source: Table 6 · p. 14, Comparison with SSL Baselines · p. 14

5.4 Effect of WiFi-JEPA Pre-training

  • WiFi-JEPA outperforms the reported baselines in MPJPE and PCK, with the comparison using a shared ViT backbone, PETR-style decoder, and identical fine-tuning.Table 6 specifies pre-training on approximately 90K real frames for 100 epochs.Source: Comparison with SSL Baselines · p. 14, Table 6 · p. 14

Ablation Studies

The ablations compare masking strategies and CSI-specific tokenization against alternatives, using matched training conditions to examine their effects on downstream pose estimation.

Ablation Studies

  • The masking comparison covers random patches, multi-block rectangles, temporal rows, and entire receiver–antenna columns.The variants are pre-trained on approximately 90K real frames for 100 epochs before fine-tuning.Source: Ablation Studies · p. 14, Table 7 · p. 14
  • 97.1 mm MPJPE and 93.0% PCK@50 are the best reported masking results; link masking beats random masking by 8.04 mm and 4.52 pp, respectively.All masking variants use the same backbone and schedule and are pre-trained on approximately 90K real frames for 100 epochs.Source: Ablation Studies · pp. 14–15, Table 7 · p. 14
  • CSI-specific tokenization reduces MPJPE from 111.85 to 97.1 mm and PA-MPJPE by 3.78 mm versus a 2D spectrogram baseline.The spectrogram flattens CSI into a 1 × 60 × 180 image; CSI-specific tokenization uses 60 × 20 × 9.Source: Ablation Studies · pp. 14–15

6 Conclusion

WiFi-JEPA improves WiFi-CSI 3D pose estimation across single- and multi-person settings, while its evaluation remains limited to one fixed-hardware dataset. Its conclusion presents CSI-native self-supervised learning as a promising direction beyond pose estimation.

6 Conclusion

  • Across single- and multi-person PiW3D 3D pose estimation, WiFi-JEPA sets a new state of the art and nearly halves cross-environment error.It consistently improves over training from scratch, whereas four vision-native SSL objectives degrade performance.Source: §6 · p. 15
  • WiFi-JEPA predicts masked latent embeddings on factored (C, T, L) CSI, using structure-aware tokenization and link masking to capture cross-link spatial correlations.Ray-tracing simulation supplies scalable pre-training data without annotation.Source: §6 · p. 15
  • Simulated CSI from simple moving primitives offers pre-training value comparable to real data, suggesting channel-dynamics diversity matters more than scatterer geometric realism.Source: §6 · p. 15
  • WiFi-JEPA improves global joint localization more than relative joint configuration, with PA-MPJPE staying close to the from-scratch baseline.Cross-environment error remains substantially reduced but far higher than same-environment performance.Source: §6 · p. 15
  • Evaluation is confined to one dataset with fixed hardware, leaving generalization across antenna configurations and frequency bands open.The paper describes CSI-native self-supervised learning as a promising direction for WiFi sensing beyond pose estimation.Source: §6 · p. 15
Loading 2607.11064v1…