Kinematics-Induced Multimodal 3D Human Pose Estimation with Subject-Level Privacy

• RGB, LiDAR, and radar provide complementary cues, but their differing feature distributions and noise make it challenging to align them while retaining modality-specific evidence.Source: §1 · p. 1
• Correlated records from the same subject can make record-level privacy guarantees understate individual risk, while DP-SGD can substantially reduce utility.Source: §1 · p. 1
• Prior work has largely studied multimodal pose estimation and privacy separately, leaving subject-level protection and leakage beyond aggregate membership accuracy open.Source: §1 · pp. 1–2
• The proposed fusion module aligns modality-specific joints, injects skeletal structure, and adaptively aggregates sensor evidence, outperforming individual modalities and achieving state-of-the-art performance.Source: §1 · p. 2
• The paper pairs a black-box subject membership attack with empirical pointwise maximal leakage to characterize how individual attack outcomes change membership beliefs.Source: §1 · p. 2
• Action Temporal Stratification supports subject-level DP by weighting within-subject sampling to cover actions and time while preserving an unbiased subject gradient.Source: §1 · p. 2

2.1 3D HPE using multimodal sensors

• RGB-based pose methods can be sensitive to occlusion, illumination, viewpoint, resolution, and scene clutter because they do not directly observe 3D geometry.Source: §2.1 · p. 2
• LiDAR contributes explicit 3D geometry and illumination invariance, while sparse or incomplete point clouds and coarse, noisy radar observations constrain both modalities.Source: §2.1 · p. 2
• Using multiple sensors at inference increases hardware cost, calibration complexity, synchronization overhead, and deployment constraints; teacher-student designs instead use one modality at inference.Source: §2.1 · p. 2

2.2 Privacy-preservation in computer vision

• Standard DP-SGD can reduce visual prediction utility, with rare-pose imbalance, high dimensionality, loss smoothness, and clipping calibration affecting performance.Source: §2.2 · p. 2
• Record-level DP is insufficient for strongly correlated, multi-sample subject data; LiDAR and radar motion sequences therefore motivate user-level protection and correlated sampling.Source: §2.2 · pp. 2–3

3.1 Overall Architecture

• DSTFormer-based RGB and PointTransformerV2-based LiDAR and Radar encoders feed 17 joint tokens into a regression head for final 3D pose prediction.The architecture is described in Figure 1.Source: §3.1 · p. 3

3.2 Fusion Module

• A three-stage fusion module aligns joint features across sensors, maintains modality-specific streams and shared tokens for skeletal refinement, then adaptively weights evidence per joint and updates shared tokens.Kinematic refinement combines fixed skeletal adjacency with learned joint dependencies; fusion weights vary across joints and samples.Source: §3.2 · pp. 3–4

3.3 Action Temporal Stratified Subject-Level Differential Privacy

• Subject-level privacy treats all pose windows from one person as a single privacy unit, aggregates sampled windows before clipping once per subject, and retains the same outer privacy accounting under ATS.The subject contribution remains bounded by the clipping threshold, while outer subject sampling and Gaussian noise remain unchanged.Source: §3.3 · pp. 5, 7
• Action Temporal Stratification samples a limited number of windows across actions and chronological bins, using population weights to preserve the intended subject-mean gradient in expectation.The motivation is to avoid overrepresenting frequent actions or short temporal regions under a finite window budget.Source: §3.3 · p. 6

3.4 Subject Level Membership Inference and Pointwise Maximal Leakage

• The membership audit uses prediction-only access to a fixed model and query windows with pose annotations; lower MPJPE yields a higher subject score and stronger membership evidence.The study uses subject-level ROC-AUC as its threshold-independent metric and cross-fitted thresholds for balanced-accuracy evaluation.Source: §3.4 · p. 7
• Pointwise maximal leakage estimates how a particular score outcome shifts belief under a balanced membership prior, with zero indicating no change and log2 complete disclosure.The empirical maximum is taken only over bins containing an observed subject; this estimate is not a formal guarantee.Source: §3.4 · pp. 7–8

4.1.1 Effect of Multimodal Fusion

• 5.46 mm, 10.81 mm, and 11.05 mm are the MPJPE reductions over the strongest unimodal model under random, cross-environment, and cross-subject splits, respectively.The fusion model achieves the lowest MPJPE across all three splits under Protocol 3.Source: §4.1.1 · p. 8

4.1.2 ATS Subject Level DP and Attack inference

• At ϵ = 1, membership ROC-AUC is near chance and empirical PML is zero, but pose accuracy incurs a severe utility cost.As ϵ increases, pose accuracy improves alongside attack distinguishability.Source: §4.1.2 · p. 8
• The non-private model (ϵ = ∞) has the best MPJPE and strongest membership signal, with 7 of 8 held-out subjects in the higher-error half of attack scores.With only 8 held-out subjects, only the non-private model and ϵ = 200 are individually distinguishable from chance.Source: §4.1.2 · p. 8
• Empirical PML changes more coarsely than ROC-AUC because it relies on discretized score outcomes.Table 2 reports privacy-utility and empirical leakage under Protocol 3's cross-subject setting; its rows correspond to privacy budgets and a non-DP baseline.Source: §4.1.2 · p. 8, Table 2 · p. 9

5 Conclusion

• The framework combines kinematics-induced fusion for multimodal 3D HPE with subject-level DP training and empirical membership inference.Action Temporal Stratification constructs population-weighted subject gradients, while membership inference and PML provide complementary evidence about model privacy.Source: §5 · p. 9
• Fusion improves pose accuracy over individual modalities, while subject-level DP degrades utility, particularly at stricter privacy budgets.The evaluation is limited to the predefined MM-Fi subject split with eight non-member subjects; future work should study private adaptation and broader subject cohorts.Source: §5 · p. 9

6.1 Experimental Setup

• Experiments use RGB, LiDAR, and radar from MM-Fi, which contains over 320K synchronized frames across five modalities, 40 subjects, and 27 action categories.Evaluation uses the dataset's random, cross-subject, and cross-environment splits across all three protocols; MPJPE and PA-MPJPE are reported.Source: §6.1 · p. 12

6.2 Additional results of fusion method on Protocol 1 and 2

• Table 3 compares Protocol 1 daily activities and Protocol 2 rehabilitation activities using MPJPE and PA-MPJPE, where lower values indicate better performance.The supplied passage gives the comparison scope and metrics, not the table's cell values.Source: Table 3 · p. 13

6.3 Bimodal Ablations

• Table 4 compares RGB+LiDAR, RGB+Radar, and LiDAR+Radar across three activity protocols and evaluation splits using MPJPE and PA-MPJPE, where lower is better.The supplied passage does not report the values for the combinations.Source: Table 4 · p. 15

6.4 Comparison with BeV fusion

• Table 5 compares the BEV-based fusion baseline [57] with LiDAR+Radar fusion on Protocol 3 across random, cross-environment, and cross-subject splits using MPJPE, where lower is better.The supplied passage states the comparison scope but not the results.Source: Table 5 · p. 16