PaperlayerView exact revision on arXiv

Source-linked AI summary

HuC-VideoMAE: Human-Centric Video Masked Autoencoding from synthetic data

Ricardo Pizarro, Roberto Valle, José M. Buenaposada, Luis M. Bergasa, Luis Baumela

arXiv:2610.08433v1cs.CV

TL;DR

Web-crawled video raises consent and biometric-privacy concerns, motivating synthetic human-motion pretraining as an alternative. HuC-VideoMAE uses keypoints and person localization to guide masked reconstruction. It improves synthetic pretraining transfer across NTU RGB+D and Toyota-Smarthome, recovering approximately 49% of the NTU RGB+D cross-view-subject gap to Kinetics without real pretraining video.

  • Problem

    Web-crawled video pretraining can use identifiable people without consent, motivating the question of whether synthetic human-motion data can support video-transformer pretraining without sacrificing downstream accuracy.

    Source: §1 · p. 2

  • Method

    HuC-VideoMAE uses synthetic-data keypoints and person-localization information to guide human-centric masked reconstruction, including identity-level keypoint masking.

    Source: §1 · pp. 2–3, §5 · p. 13

  • Results

    Across NTU RGB+D and Toyota-Smarthome, HuC-VideoMAE improves over standard BEDLAM2.0 VideoMAE; on NTU RGB+D cross-view-subject it recovers approximately 49% of the gap to Kinetics pretraining.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The results indicate that narrowing the synthetic-to-real pretraining gap depends on objective design that exploits annotations available in synthetic data, as well as corpus quality and scale.

    Source: §5 · p. 13

    Comparisons with other synthetic-to-real methods require caution because backbone architecture, pretraining data, and supervision strategy differ.

    Source: §4.6 · p. 11

Abstract

from arXiv · show

Modern action recognition models rely on video transformers pretrained on massive collections of web-crawled videos, such as Kinetics-700. However, the use of such data raises ethical concerns, as subjects' consent is typically not obtained. Recent high-quality synthetic video datasets generated from motion-capture data, such as BEDLAM2.0, offer a promising ethical alternative. In this work, we investigate self-supervised pretraining of video transformers on synthetic human-motion datasets. We first show that directly applying the standard VideoMAE masking strategy leads to substantially worse performance than pretraining on Kinetics. To address this limitation, we propose a human-centric masking scheme that leverages body keypoints and person bounding box regions. Our approach encourages the model to focus on the structure and dynamics of human motion during pretraining. Experiments on NTU RGB+D and Toyota-Smarthome demonstrate that our method significantly outperforms standard VideoMAE pretraining on synthetic data, closing 49% of the gap to Kinetics pretraining on NTU RGB+D cross-view-subject without using a single real frame during pretraining. To promote the use of ethical action recognition models, we will publicly release our pretrained models.

1 Introduction

Web-crawled video pretraining raises consent and biometric-privacy concerns, while synthetic motion-capture humans offer a privacy-preserving alternative. HuC-VideoMAE adapts masking to human structure and improves transfer on NTU RGB+D and Toyota-Smarthome without real pretraining frames.

  • Web-crawled pretraining videos can depict identifiable people who did not consent to use of their likeness, gait, or behaviour.The paper notes that identity can be recovered from appearance or gait even in de-identified data.Source: §1 · p. 2
  • Synthetic motion-capture datasets provide photorealistic clothed avatars without depicting real people, avoiding consent and biometric-leakage risks by construction.The paper identifies control over appearance, viewpoint, and scene as properties of these datasets.Source: §1 · p. 2
  • Standard VideoMAE masking transfers substantially worse than Kinetics pretraining on synthetic video, as background and rendered texture can dominate reconstruction.At a 90% masking ratio, random tube masking can leave the actor nearly all visible or hidden by chance, encouraging shortcuts on limited synthetic data.Source: §1 · pp. 2–3
  • HuC-VideoMAE uses synthetic keypoints and person boxes to prioritize body regions while keeping sparse joints visible as structural anchors.Selected keypoint identities remain masked throughout a clip, concentrating reconstruction on body motion.Source: §1 · pp. 2–3
  • 85.8% cross-view-subject average accuracy rises from 80.1%, recovering 49% of the gap to Kinetics-pretrained 91.7%.On Toyota-Smarthome, it exceeds the Kinetics-pretrained VideoMAEv2 baseline on CV1 and is within 4 percentage points on CV2.Source: §1 · p. 3
  • The work investigates synthetic motion-capture pretraining as a privacy-preserving alternative and finds standard VideoMAE masking less effective in that setting.Source: §1 · p. 3

2 Related work

Prior work explores masked video reconstruction, content-aware masking, and synthetic-human datasets for action recognition. This paper distinguishes its approach by designing self-supervised masking around the structural annotations available in synthetic data.

  • Prior content-aware masking methods guide masks using optical flow, synthetic-object trajectories, or facial regions, while masked video models reconstruct heavily masked inputs.The cited examples include MGMAE, SMILE, and MARLIN.Source: §2 · p. 3
  • Many prior masking approaches retain web-sourced video, whereas synthetic data can provide ground-truth signals for selecting informative regions without noisy optical-flow or facial-segmentation estimates.Source: §2 · pp. 3–4
  • Earlier synthetic-human work includes PHAV, SURREAL, SURREACT, AMARV, and BEDLAM2.0, with prior applications spanning cross-view recognition and synthetic-to-real pretraining.Source: §2 · p. 4
  • The paper asks how to make ethically sourced synthetic pretraining competitive by designing the self-supervised objective around human-centric structure.Its figure assigns patches to keypoint, body, and background regions, with keypoints generally receiving the lowest masking scores.Source: §2 · p. 4, Figure 1 · p. 4

3 Human centric video masked autoencoder pretraining

HuC-VideoMAE retains the VideoMAE masked-autoencoding framework but replaces uniform masking with a policy based on synthetic keypoints and person boxes. The policy prioritizes background and body patches for masking while preserving sparse visible joints and consistently hiding selected joint identities across a clip.

3 Human centric video masked autoencoder pretraining

  • HuC-VideoMAE builds on VideoMAE masked autoencoding, reconstructing many spatiotemporal patches from a smaller set of visible patches.Its contribution is a masking policy using synthetic per-frame keypoints and person boxes to focus reconstruction on the human actor.Source: §3 · p. 5

3.1 Body-keypoint-guided masking

  • After crop-and-resize augmentation, keypoints and boxes are mapped to the patch grid; background patches score highest, body patches intermediate, and unmasked keypoints lowest.The highest-scoring patches are selected for masking.Source: §3.1 · pp. 5–6
  • A fraction of keypoint identities is selected once per clip and forced to remain masked in every frame, preventing the decoder from copying a joint visible in an adjacent frame.The per-frame masked count is fixed at K = r·Np, where r is the target masking ratio and Np is the number of patches.Source: §3.1 · pp. 5–6
  • The policy directs reconstruction toward masked body regions inferred from visible joint structure and motion rather than background regularities.Source: §3.1 · pp. 5–6

4 Experiments

Experiments pretrain VideoMAE on synthetic BEDLAM2.0 and fine-tune on NTU RGB+D and Toyota-Smarthome, comparing human-centric masking with random masking and real-data references. They assess masking-policy gains, synthetic-corpus effects, and synthetic-to-real transfer.

4.1 Datasets

  • BEDLAM2.0 provides synthetic motion sequences with human annotations; NTU tests front-view training across three views, while Toyota uses CV1/CV2 cross-view protocols.The benchmarks report NTU accuracy and Toyota mean class accuracy (mCA).Source: §4.1 · pp. 7–8

4.2 Implementation details

  • Pretraining uses 16-frame clips, 90% patch masking, region-specific masking scores, and 20% clip-level keypoint-identity masking for 200 epochs.The implementation uses ViT-B and MViTv2-S backbones; MViTv2 masking is expanded to macro-patches to preserve its regular grid structure.Source: §4.2 · pp. 8–9

4.3 Training and Evaluation Procedure

  • The evaluation compares Kinetics VideoMAE, synthetic BEDLAM2.0 VideoMAE with random tube masking, and HuC-VideoMAE under supervised downstream fine-tuning.The two synthetic variants share data, schedule, and hyperparameters, isolating masking policy in their comparison.Source: §4.3 · p. 9, §4.4 · p. 9

4.4 Effect of the masking policy

  • 5.7 of the 11.6-point NTU RGB+D cross-view-subject accuracy gap is recovered by HuC-VideoMAE, closing 49% without real pretraining frames.With random tube masking, synthetic BEDLAM2.0 pretraining scores 80.1 versus 91.7 for Kinetics; HuC-VideoMAE scores 85.8.Source: §4.4 · p. 9, Table 1 · p. 10

4.5 Effect of the synthetic corpus

  • BEDLAM2.0 outperforms AMARV under random tube masking, averaging 80.1 vs. 55.0 on NTU and 56.5 vs. 42.3 on Toyota-Smarthome.The authors attribute the difference to BEDLAM2.0’s greater body diversity, realistic camera motion, and rendering quality, rather than corpus scale alone.Source: Table 2 · p. 10, §4.5 · p. 10

4.6 Comparison with the state of the art

  • NTU: 86.6/85.9/84.8 vs. Kinetics’ 92.2/91.8/91.2; Toyota CV1: 56.7 vs. 55.20 mCA, while HuC CV2 reaches 63.70 vs. 67.68.The NTU values are ViT-B accuracy across 0°/45°/90°; the Toyota values are VideoMAEv2-base mCA on CV1/CV2.Source: Table 3 · p. 11, §4.6 · pp. 11–12, Table 4 · p. 12
  • HuC-VideoMAE has the highest average among the listed synthetic-to-real methods, but comparisons vary in backbone, pretraining data, and supervision.It exceeds several fully supervised Kinetics-pretrained methods, while π-ViT and PO-GUISE score higher using additional inductive biases and pose-based modeling.Source: §4.6 · pp. 11–13

5 Conclusion

HuC-VideoMAE uses human-centric masking to improve synthetic human-motion pretraining and transfer to real benchmarks. Results suggest both synthetic-corpus quality and objectives that exploit synthetic annotations matter for narrowing the gap to real-video pretraining.

5 Conclusion

  • HuC-VideoMAE addresses the gap by using synthetic datasets’ body keypoints and person-localization information for human-centric masking.The strategy is intended to encourage motion- and pose-aware representations.Source: §5 · p. 13
  • HuC-VideoMAE improves over standard synthetic VideoMAE across NTU RGB+D and Toyota-Smarthome, recovering approximately 49% of the NTU cross-view-subject gap to Kinetics while remaining competitive on Toyota-Smarthome.It exceeds the Kinetics-pretrained VideoMAEv2 baseline on Toyota-Smarthome CV1 and remains within 4 percentage points on CV2, without using real video during pretraining.Source: §5 · p. 13
  • Narrowing the synthetic-to-real pretraining gap depends on both synthetic-corpus quality and scale and objectives that exploit annotations unique to synthetic data.The authors frame this as motivation for further human-centric self-supervised learning with ethically sourced data.Source: §5 · p. 13
Loading 2610.08433v1…