PaperlayerView exact revision on arXiv

Source-linked AI summary

PT$^3$: Prototype-Guided Self-Supervised Pretraining for Generalizable mmWave Human Point Cloud Representation

Ziyang Liu, Kieren Yu, Xiao Yang, Xin Yang, Kaishun Wu

arXiv:2610.38551v1cs.AI

TL;DR

Sparse, unstable mmWave detections and task-specific or limited prior pretraining leave a need for reusable representations across sensing tasks and sources. PT3 represents local states as shared-prototype compositions, learning temporal evolution and observation consistency. Its frozen encoder outperforms compared pretrained baselines across four downstream tasks and achieves competitive or superior performance to listed task-specific methods.

  • Problem

    Existing methods are often task- and dataset-specific, while recent radar pretraining targets limited tasks or relies on paired modalities; mmWave detections also lack stable body-part correspondences.

    Source: §1 · pp. 1–2

  • Method

    PT3 uses body-agnostic spatial regions to represent local human states as weighted combinations of shared prototypes, learning their evolution autoregressively and stabilizing them with observation consistency.

    Source: §1 · p. 2

  • Results

    PT3 outperforms compared pretrained baselines across four downstream tasks and achieves competitive or superior performance to listed task-specific methods.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The results support learning transferable human representations from unstable mmWave observations using unlabeled radar sequences and a shared frozen encoder.

    Source: §6 · p. 11

    Because detections lack stable correspondences to body parts, the representation is defined over consistent spatial references rather than joints or anatomical parts.

    Source: §1 · p. 2

Abstract

from arXiv · show

Multi-source millimeter-wave (mmWave) observations offer an opportunity to learn human representations reusable across sensing tasks. However, sparse detections and multipath effects make these observations unstable, challenging existing point cloud pretraining methods. We propose PT$^3$, a prototype-guided self-supervised pretraining framework for mmWave human point clouds. PT$^3$ represents local states in spatial regions as sparse compositions of shared prototypes and learns their temporal evolution through autoregressive prediction. Observation consistency further stabilizes representations under radar perturbations. After pretraining on unlabeled sequences from multiple sources, the same frozen encoder supports action classification, dense point generation, joint pose estimation, and future motion prediction on datasets not used for pretraining. Across these tasks, PT$^3$ outperforms the compared pretraining baselines and achieves competitive or superior performance to task-specific models.

1 Introduction

Existing mmWave pretraining does not yet provide a unified, broadly evaluated representation across sources and sensing tasks. PT3 models local human states with shared prototypes and learns their temporal evolution and consistency; the reported results outperform compared pretraining baselines and are competitive with or superior to task-specific models.

1 Introduction

  • Existing methods commonly target individual tasks and datasets with task-specific labels, while recent radar pretraining targets limited tasks or relies on paired modalities.Source: §1 · p. 1
  • mmWave points are radar detections rather than surface samples, and viewing angle, multipath, and limited angular resolution make them sparse, incomplete, and inconsistent.A point change may reflect sensing rather than human motion, motivating targets based on underlying human states.Source: §1 · p. 2
  • The proposed representation describes local states as weighted combinations of shared prototypes across body-agnostic spatial regions.PT3 forecasts composition changes and uses observation consistency to stabilize representations under point dropout and spurious echoes.Source: §1 · p. 2
  • The paper evaluates a frozen PT3 encoder on unseen datasets across action classification, dense point generation, joint pose estimation, and future motion prediction.Source: §1 · p. 2
  • PT3 outperforms the compared pretraining baselines and achieves competitive or superior performance to task-specific models.Source: §1 · p. 3

2 Related Work

Related work spans mmWave human-sensing datasets and tasks, general point-cloud pretraining, and radar representation learning. The paper identifies unified self-supervised pretraining across mmWave sources and devices, together with broad shared-encoder evaluation, as underexplored.

Tasks and Datasets in mmWave Human Sensing

  • mmWave human sensing has expanded from activity and gesture recognition to pose estimation, body reconstruction, and motion modeling, with datasets covering more subjects, scenes, and tasks.Source: §2 · p. 3

General Point Cloud Pretraining and Representation Learning

  • General point-cloud baselines include masked modeling, autoregressive point-patch generation, and self-distillation on large-scale data.The cited examples are Point-BERT, Point-MAE, Point-M2AE, PointGPT, and Sonata.Source: §2 · p. 3

Representation Learning for mmWave Human Sensing

  • Radar representation work includes self-supervision, cross-modal alignment, semantic-token alignment, and wireless-signal pretraining, but unified multi-source pretraining and broad shared-encoder evaluation remain underexplored.Source: §2 · p. 3

3 Methodology

PT3 represents local human states behind unstable radar detections using fixed spatial regions and shared latent prototypes. It learns regional compositions, their temporal evolution, and consistency across perturbed observations.

3.1 From Radar Points to Spatial Slots

  • A frame adapter compresses variable detections into representative features, while FiLM conditions the point encoder on device-specific sensing characteristics.The conditioning inputs are field-of-view bounds, spatial resolution, and validity indicators.Source: §3.1 · p. 4
  • The encoder maps radar observations into fixed spatial slots, whose consistent indices make local states comparable across time despite changing slot contents.Slots aggregate nearby returns through overlapping soft routing around fixed anchors; a null slot absorbs returns far from the grid.Source: §3.1 · pp. 4–5

3.2 Compositional Motion Modeling

  • Each spatial region represents its local state as a weighted composition of a few learnable prototypes shared across all regions.This composition avoids assigning a local state to a single prototype and does not require a point-cloud reconstruction decoder.Source: §3.2 · p. 5
  • A causal group-level Transformer predicts future regional prototype distributions from historical compositions, capturing temporal evolution and coordination across regions.The causal mask prevents predictions from accessing observations at the prediction time or later; EMA-encoder assignments provide future targets.Source: §3.2 · p. 5, §3.3 · p. 6

3.3 Pre-Training Objectives

  • A separate balanced-assignment loss trains the shared prototype bank, encouraging reusable latent elements and limiting concentration on a few prototypes.Sinkhorn balancing supplies assignment targets; this loss updates only the prototype bank, separately from temporal prediction.Source: §3.3 · p. 6
  • Observation consistency aligns perturbed and clean spatial representations using support from the clean view, maintaining supervision when perturbations remove detections.Perturbations include local point dropout and added radially shifted spurious echoes; the objective uses weighted SmoothL1 loss.Source: §3.3 · pp. 6–7

4 Experiment

The experiments evaluate one frozen pretrained encoder on four downstream tasks using data sources excluded from pretraining. PT3 outperforms the compared pretrained baselines across the tasks, with mixed results against task-specific methods on future motion prediction.

4.1 Datasets

  • The evaluation covers action classification, dense point generation, joint pose estimation, and future motion prediction, using the specified dataset protocols and splits.Dense point generation uses the mmBody split and excludes smoke and occlusion scenes; pose estimation follows MM-Fi protocols, while future motion prediction follows mmPred’s cross-subject split on MM-Fi.Source: §4.1 · pp. 7–8

4.2 Experiment Settings

  • Pretraining uses unlabeled point-cloud sequences from four sources, while all downstream evaluation data are excluded from pretraining.The four sources contain approximately 1.29M raw frames; preprocessing resamples sequences to 12 Hz, and training uses no action, skeleton, or mesh labels.Source: §4.1 · p. 7, §4.2 · p. 8
  • All four downstream tasks share the same frozen pretrained encoder and use lightweight task-specific decoders.The comparison includes general point-cloud pretrained encoders, Sonata, and the radar-specific Wave2Body tokenizer, with the same split and prediction head within each task.Source: §4.2 · p. 8

4.3 Experiment Results

  • PT3 outperforms every compared pretrained baseline across four tasks and surpasses task-specific supervised methods on all reported action-classification, dense-generation, and pose-estimation metrics.For future motion prediction, it has lower ADE and FDE than mmPred on Pickup, Throwing, Kicking, and Bowing, while mmPred is stronger on Chest Expand, Side Twist, and Raise Hand.Source: §4.3 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8
  • PT3 reaches 81.35% and 85.23% accuracy on UniMM-HAR under C-Sub and C-Set, respectively.On mmBody, its CD/EMD are 0.91/4.70; on MM-Fi, its Cross-Environment MPJPE is 77.43 mm.Source: §4.3 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8

5 Ablation and Analysis

The ablation study compares pretraining variants across four downstream tasks, varying observation consistency, autoregressive prediction, and prototype modeling. The section also presents prototype-composition comparisons across actions and subjects and examines hyperparameter sensitivity.

Ablation

  • The ablation compares variants that remove observation consistency, autoregressive prediction, or prototype modeling, with all variants pretrained from scratch on the same data and protocols.Removing prototype modeling also removes the prototype bank and its update signal, replacing it with direct prediction of continuous slot features.Source: §5 · p. 9, Table 6 · p. 10

Ablation on Observation Consistency

  • In the observation-consistency ablation, C-Sub accuracy rises slightly from 81.35%, but the supplied passage does not state the comparison value.Source: §5 · p. 9

Prototype-composition analysis

  • Figure 3 compares prototype-composition JS divergence between actions averaged over subjects and between subjects averaged over actions; darker cells indicate larger differences.The figure’s two views distinguish action-related differences from subject-related differences.Source: Figure 3 · p. 10

Hyperparameter sensitivity

  • The study analyzes sensitivity to the number of spatial slots S, top-k prototype selection, and prototype bank size K.Source: §5 · p. 10

6 Conclusion

The conclusion presents PT3 as a framework that organizes unstable mmWave detections into spatial slots and shared prototype compositions, then learns their temporal evolution. The paper reports transfer across four downstream tasks and states that it uses only publicly available datasets, with no newly collected human-subject data.

6 Conclusion

  • PT3 learns shared prototype compositions for local states and their temporal evolution from multi-source mmWave point clouds.The framework organizes detections into spatial slots and uses autoregressive prediction and observation consistency under radar perturbations.Source: §6 · pp. 10–11, A Extended Related Work · p. 17
  • After pretraining on unlabeled radar sequences, the same frozen encoder outperforms compared pretrained baselines across four downstream tasks and is competitive with or superior to listed task-specific supervised methods.The conclusion says ablations support the three design components and prototype-composition analysis finds similar assignment patterns for the same action across subjects.Source: §6 · p. 11

Ethics Statement

  • The study uses only publicly available mmWave human-sensing datasets processed by their original providers and collects no new human-subject data.Source: Ethics Statement · p. 11

A Extended Related Work

  • Prior radar pretraining studies mainly evaluate pose recovery or use paired modalities, whereas PT3 learns local states and their temporal evolution from sparse point clouds.Wave2Body and MAEPose mainly evaluate pose recovery; WiSwiss uses a time–range–angle tensor, and its public semantic-deduplication encoder uses class labels to select negative pairs.Source: A Extended Related Work · p. 16
  • Other sensing approaches use task-specific supervision or cross-modal information, including supervised feature extractors, pose labels, motion features, or text.The paper contrasts these approaches with learning from radar observations without action, pose, mesh, or text supervision.Source: A Extended Related Work · pp. 16–17

B Dataset Description

The section describes datasets spanning radar-only and synchronized multimodal human-sensing collections, then outlines PT3’s model components and points to its configuration and pretraining hyperparameters.

B Dataset Description

  • M4Human, milliFlow, MARS, and MiliPoint provide varied sensing streams or radar point clouds, with differing subject, action, and acquisition specifications.milliFlow includes three synchronized streams and three collection sites; MiliPoint releases radar point clouds, while MARS pairs them with Kinect skeleton measurements.Source: B Dataset Description · p. 17
  • DGUHA pairs radar sequences with Kinect skeleton annotations, mmBody provides radar, RGB, and depth streams, and mRI includes radar, RGB, depth, and inertial signals.The mRI collection also includes 2D keypoint and 3D joint annotations.Source: B Dataset Description · p. 17
  • MM-Fi provides six synchronized sensing streams, while RadHAR is radar-only and UniMM-HAR combines radar data from RadHAR, mRI, and MM-Fi.MM-Fi covers 40 subjects and 27 actions; RadHAR has two subjects and five actions, and UniMM-HAR combines 62 source subjects and 33 unified action categories.Source: B Dataset Description · p. 18

C.1 Model Implementation

  • PT3 comprises a shared observation encoder, a prototype bank, and an autoregressive prediction module.The encoder uses a Frame Adapter, point cloud encoder, device-conditioned FiLM, spatial readout, and shared FFN, in that order.Source: §C.1 · p. 19
  • Table 7 summarizes PT3’s model configuration.The passage does not provide the table’s configuration values.Source: Table 7 · p. 19

C.2 Hyperparameters

  • Table 8 summarizes the pretraining hyperparameters.The passage does not provide the table’s hyperparameter values.Source: Table 8 · p. 20

D Downstream Evaluation Metrics

The section defines task-specific metrics for downstream evaluation and describes the compared pretraining baselines. It also introduces the action-classification section.

D Downstream Evaluation Metrics

  • Accuracy is the percentage of action-classification test samples whose predicted class matches the ground truth, while weighted F1 weights class-wise F1 by test-sample counts.Accuracy is reported for UniMM-HAR and DGUHA; weighted F1 is reported for DGUHA.Source: D Downstream Evaluation Metrics · p. 20
  • CD measures bidirectional nearest-point squared-distance discrepancy, whereas EMD uses one-to-one squared-distance matching between generated and target point clouds.The point-cloud metrics are computed after centering and displayed as 100 times their raw m2 values.Source: D Downstream Evaluation Metrics · p. 20
  • MPJPE averages Euclidean joint-position error, PA-MPJPE applies that error after pose alignment, and ADE averages frame-wise motion error over ten predicted frames.MPJPE is reported in millimeters without per-pose alignment; PA-MPJPE aligns translation, rotation, and scale, while ADE uses translation-only centroid alignment.Source: D Downstream Evaluation Metrics · pp. 20–21
  • FDE uses future-pose error at the tenth predicted frame, with ADE and FDE minimized independently over ten generated futures for each test window.The cited passage describes the final-frame measure but does not spell out its acronym.Source: D Downstream Evaluation Metrics · p. 21

E Pretraining Baselines

  • The comparison includes Point-MAE, Point-BERT, Point-M2AE, PointGPT-S, Sonata, and Wave2Body, with the four general point-cloud baselines pretrained on the same unlabeled data as PT3.The encoders are frozen for downstream evaluation; Wave2Body is the mmWave-specific method described among the baselines.Source: E Pretraining Baselines · p. 21

F Action Classification

  • The following section describes the dataset, evaluation protocols, and implementation details for action classification.Source: F Action Classification · p. 21

F.1 Dataset and Protocols

The section describes UniMM-HAR’s action-classification protocols and shared frozen-feature evaluation procedure, then introduces the dense point-generation section.

F.1 Dataset and Protocols

  • UniMM-HAR unifies radar point clouds from RadHAR, mRI, and MM-Fi into 33 action categories and uses cross-subject and cross-scene protocols.Windows from the same original sequence do not cross data splits, and the two protocols are distinct partitioning views rather than combined sample counts.Source: §F.1 · p. 21
  • The supervised action-classification comparisons use published RadHAR, FastHAR, and DAP-Net results under the same evaluation splits.Table 9 summarizes the two UniMM-HAR protocols and their sample counts.Source: §F.1 · p. 21, Table 9 · p. 22

F.2 Implementation

  • All pretrained encoders are frozen, and features from three consecutive frames are fused before a shared linear Ridge-regression probe is fitted.Probe regularization is selected by two-fold environment-wise cross-validation within the official training set.Source: §F.2 · p. 22

G Dense Point Generation

G.1 Dataset and Protocols

The section specifies the mmBody split and shared point-generation decoder, then outlines the joint-pose dataset, protocols, and head; it also introduces future motion prediction.

G.1 Dataset and Protocols

  • mmBody supplies synchronized radar, depth, and RGB data, and the evaluation split excludes smoke and occlusion test scenarios.The test set contains 20,207 frames from lab1, lab2, furnished, poor_lighting, and rain; supervised comparisons use published results under the same split.Source: §G.1 · p. 22

G.2 Implementation

  • Frozen features feed a shared projection and residual convolutional decoder that reconstructs 512 unordered 3D points from the corresponding depth-camera point cloud.The decoder is trained with the objective L = CD + 0.19 EMD for 25 epochs, and final EMA weights are used for evaluation.Source: §G.2 · pp. 22–23

H Joint Pose Estimation

H.1 Dataset and Protocols

  • MM-Fi provides 17-keypoint 3D skeleton annotations across 27 daily activities and four environments, evaluated with Random, Cross-Subject, and Cross-Environment protocols.The task regresses current-frame joint coordinates; reported MPJPE and PA-MPJPE are in millimeters, and supervised comparisons use results from the mmDiff paper.Source: §H.1 · p. 23

H.2 Implementation

  • The joint-pose evaluation freezes each encoder and trains only a TCN head on extracted features.The shared head projects features to 128 dimensions, uses four residual causal TCN blocks, and outputs joint coordinates.Source: §H.2 · p. 23

I Future Motion Prediction

I.1 Dataset and Protocols

The evaluation describes the MM-Fi future-motion protocol, frozen-encoder downstream readouts, and DGUHA action-classification results. On DGUHA, AHC reaches 94.41% accuracy and 94.52% weighted F1.

I.1 Dataset and Protocols

  • MM-Fi future-motion evaluation uses 27 activities from 40 subjects across four environments under a cross-subject protocol.Methods observe eight historical radar frames and predict ten future frames of 16 non-root joint trajectories; lower mean-aligned ADE and FDE indicate better performance.Source: §I.1 · p. 24

I.2 Implementation

  • For future motion, a TCN decoder predicts conditions for the frozen official diffusion model, with only TCN parameters updated during its final optimization.The TCN uses four residual causal convolution blocks; the supervised comparison results are taken from the mmPred paper under the same MM-Fi protocol.Source: §I.2 · p. 24, §I.1 · p. 24

J.1 Action classification on DGUHA.

  • On DGUHA, AHC achieves 94.41% accuracy and 94.52% weighted F1 with a frozen PT3 encoder and Ridge-regression linear probe.The table caption identifies ZP, GN, and AHC as the dataset’s three processed point-cloud inputs.Source: §J.1 · p. 24, Table 10 · p. 24

J.2 Hyperparameter Analysis

Hyperparameter analysis finds task-dependent sensitivity to prototype selection, bank size, and spatial slots, while composition analyses show action-dependent patterns and cross-subject similarity. The visualizations include pose and future-motion predictions, and the paper identifies edge deployment and radar-based motion world models as future directions.

J.2 Hyperparameter Analysis

  • k = 4 gives the most consistent results; smaller k generally hurts action classification, dense point generation, and future motion, while larger k yields no consistent gain.The authors suggest too few prototypes may limit each slot’s capacity, while too many may make assignments less selective.Source: §J.2 · p. 25, Figure 4 · p. 25
  • K = 128 slightly improves future motion prediction but lowers action classification, dense point generation, and joint pose estimation; smaller banks generally underperform K = 64.The figure reports relative performance changes against the default K = 64 across all four tasks.Source: §J.2 · p. 25, Figure 5 · p. 25
  • One or four spatial slots cause larger losses across tasks, while increasing S from 12 to 32 produces small, task-dependent changes.The comparison uses relative performance changes against the default S = 12.Source: §J.2 · p. 25, Figure 6 · p. 26

J.3 Qualitative Action Prototype Heatmap

  • Action heatmaps differ: Kick left and Pick up object show sharp slot-axis bands, whereas Bowing and Chest expansion deviate least from the subject’s mean assignment.Red indicates increased reliance on a prototype relative to the subject’s mean, blue decreased reliance, and white values near the mean.Source: §J.3 · p. 26, Figure 7 · p. 26

J.4 Prototype Composition Detailed Results

  • Action-pair JS distances are generally larger than same-action subject-pair distances, suggesting action-dependent prototype use and similar action compositions across subjects.Figure 8 compares action pairs across 10 subjects and subject pairs across 14 actions; color encodes JS divergence.Source: §J.4 · p. 27, Figure 8 · p. 27

K Visualization

  • Pose visualizations show mostly accurate predictions, with A19 less accurate; future-motion rows show input points, current and predicted poses, and reference future poses over time.The pose examples cover six actions, and the future-motion examples cover six action categories.Source: K Visualization · p. 27, Figure 9 · p. 28, Figure 10 · p. 29

L Future Work

  • Future work proposes model compression, quantization, and lightweight implementations to reduce inference latency and memory use on resource-constrained edge devices.The stated deployment direction targets low-power mmWave human-sensing platforms.Source: L Future Work · p. 27
  • A second direction is radar-based human-motion world models, using prototype compositions as structured latent states for long-horizon dynamics and conditional generation.The proposed models could generate multiple plausible trajectories and characterize uncertainty in future states.Source: L Future Work · p. 27

Related papers

Loading 2610.38551v1…