Source-linked AI summary
MASQ: Mask-Aware Spatiotemporal Quantization for Unsupervised Skeleton Action Segmentation
Xinyao Qin, Linxiang Peng, Youbao Ye, Di Yang, Jiangtao Wang
TL;DR
Unsupervised skeleton action segmentation must handle the interaction between spatial masking, discrete quantization, and temporal boundaries without dense frame labels. MASQ uses trajectory-level joint masking and visible-joint velocity regularization to separate spatial inference from temporal smoothing. Across HuGaDB, LARa, and BABEL, it consistently outperforms existing unsupervised methods, especially in Mean over Frames accuracy, while boundary jitter can remain near rapid transitions.
Problem
Dense frame-level annotations are expensive, while masking and hard quantization can create discontinuities and unstable code switching near action boundaries.
Method
MASQ combines Joint-Level Structured Dropout, which masks complete joint trajectories, with a mask-aware velocity loss applied only to visible joints.
Results
MASQ consistently outperforms existing unsupervised methods across three skeleton datasets, with substantial improvements in Mean over Frames accuracy.
Takeaways & Limitations
Separating spatial inference from temporal supervision improves representation robustness and temporal consistency for unsupervised skeleton action segmentation.
Takeaways & Limitations
Hard nearest-neighbor quantization can still cause boundary jitter near rapid transitions, limiting uniform improvements in Edit and F1.
Abstract
from arXiv · showhide
Unsupervised skeleton-based temporal action segmentation is a crucial task for understanding human behavior in long untrimmed sequences. Recent approaches often rely on discrete quantization to discover action boundaries from motion representations. However, when spatial masking is introduced for representation learning, it can introduce representation ambiguity, while discrete quantization further amplifies small fluctuations in the latent space. The interaction between these two factors often leads to unstable code switching and severe temporal jitter near action boundaries.To address these limitations, we propose a novel Mask-aware Action Spatiotemporal Quantization (MASQ) framework. Our framework decouples the conflicting tasks of spatial feature inference and temporal smoothing.In the spatial dimension, we introduce a Joint-Level Structured Dropout (JLSD) mechanism that masks the entire temporal trajectory of selected joints, to encourage the model to learn discriminative inter-joint coordination patterns. In the temporal dimension, we design a mask-aware velocity loss that enforces motion consistency only on visible joints, that prevents gradient conflicts caused by masked signals and stabilizing temporal predictions. Extensive experiments on three widely used skeleton datasets, including HuGaDB, LARa, and BABEL, demonstrate that the proposed MASQ framework significantly outperforms existing state-of-the-art unsupervised methods. In particular, our model establishes a comprehensive and substantial leading advantage in the Mean over Frames accuracy.
Introduction
MASQ addresses the instability created when spatial masking and discrete quantization interact in unsupervised skeleton action segmentation. It combines trajectory-level joint masking with visible-joint temporal smoothing and reports consistent improvements across benchmark datasets.
- Motivation: Temporal action segmentation divides long untrimmed sequences into semantic action segments without requiring prior boundary knowledge.The task supports applications including industrial monitoring, human–robot collaboration, and healthcare analysis.
- Motivation: Dense frame-level annotations make supervised segmentation expensive and time-consuming, motivating unsupervised alternatives.Existing supervised approaches require exact action labels for every frame.
- Problem: Random temporal masking creates transition-like discontinuities, while hard quantization amplifies latent fluctuations into frequent token changes.Global velocity constraints can additionally supervise joints whose features were intentionally removed.
- Method: MASQ masks complete trajectories of selected joints with JLSD and applies velocity smoothing only to visible joints.The two mechanisms encourage inter-body-part coordination learning while avoiding conflicting supervision from masked joints.
- Results: MASQ consistently outperforms existing unsupervised methods across three skeleton datasets, with substantial gains in Mean over Frames accuracy.Mean over Frames reflects frame-level segmentation quality.
Related Work
Related work spans supervised and unsupervised temporal action segmentation alongside self-supervised skeleton representation learning. MASQ targets a limitation of short-clip masking methods by distinguishing synthetic missing observations from genuine temporal transitions.
- Temporal Action Segmentation: Temporal action segmentation research models long-range dependencies using temporal convolutions, attention mechanisms, and graph convolutional networks.
- Unsupervised Segmentation: Unsupervised segmentation discovers action boundaries without temporal labels, using pseudo-labeling, optimal transport, representation learning, and clustering.
- Skeleton Representation Learning: Skeleton representation learning commonly uses contrastive sequence augmentation and masked autoencoders based on data reconstruction.These methods learn discriminative features for action recognition.
- Research Gap: Existing skeleton self-supervised methods mainly target short trimmed sequences and do not distinguish synthetic missing observations from genuine transitions in long untrimmed motion.MASQ addresses this distinction with trajectory-level masking.
Methodology
MASQ uses a joint-disentangled sequence-to-sequence architecture with structured joint masking, discrete quantization, reconstruction, commitment, and mask-aware temporal smoothing. Its design separates spatial inference from temporal regularization while preserving skeletal geometry and stabilizing visible-joint motion.
- Joint-Level Structured Dropout: JLSD masks complete temporal trajectories of selected joints, encouraging inference from long-range inter-joint coordination rather than isolated local features.The masked latent features are later partitioned into temporal patches and mapped to a learnable codebook of prototypical motion patterns.
- Joint-Disentangled TCN Autoencoder: The joint-disentangled encoder treats each joint trajectory as an independent sample before extracting temporal features with a multi-stage dilated TCN.The decoder symmetrically recovers temporal resolution and reconstructs a skeleton sequence with the original dimensions.
- Joint-Level Structured Dropout: JLSD is a training-time regularizer: inference uses complete sequences, while its masking rate controls the strength of the spatial information bottleneck.Small rates may leave local shortcuts, whereas overly large rates remove too much motion evidence; the rate is selected through validation and sensitivity analysis.
- Training Objectives: The total training objective is a weighted sum of reconstruction, commitment, and velocity losses, with hyperparameters balancing their contributions.This combines geometric reconstruction, codebook stability, and temporal regularization in end-to-end training.
- Training Objectives: The reconstruction loss compares inter-joint distance matrices rather than absolute coordinates to preserve skeletal geometry and topological consistency.The distance matrix contains pairwise Euclidean joint distances, and the objective encourages consistent limb lengths and skeletal proportions.
- Training Objectives: The mask-aware velocity loss compares first-order temporal differences only for visible joints, decoupling spatial inference from smoothing and avoiding masked-signal gradient conflicts.An indicator identifies joints not dropped in the current batch, and normalization uses the total number of valid spatio-temporal points.
Experiments
MASQ is evaluated across HuGaDB, LARa, and BABEL using frame-level and segment-level metrics, with strong semantic gains but less uniform boundary quality on complex BABEL motions.
- Evaluation setup: Experiments use HuGaDB, LARa, and three BABEL subsets, comparing MASQ with unsupervised temporal action segmentation baselines.The evaluation includes sensor-based and 3D-skeleton modalities.
- Evaluation setup: MoF, Edit Score, and F1 at 10%, 25%, and 50% measure frame assignment, action ordering, fragmentation, and boundary quality.MoF is particularly informative for unsupervised semantic discovery, while Edit and F1 expose segment coherence.
- Comparison with State of the Art: 52.3% MoF on HuGaDB surpasses the state-of-the-art baseline by 10.3 percentage points, while LARa MoF rises from 37.4% to 47.2%.The gains extend to Edit Score and all reported F1 scores on both datasets.
- Comparison with State of the Art: 46.7%, 55.8%, and 47.7% are MASQ’s highest MoF values on the three BABEL subsets, but Edit and F1 improvements are not uniform.MASQ leads all reported metrics on Subset-2 and F1@50 on Subset-3, while ASOT or SMQ remains stronger on several boundary-sensitive measures.
- Qualitative evaluation: Representative segmentations show fewer short label oscillations and better preservation of dominant action regions, while BABEL retains fragments and shifted endpoints.These residual errors are consistent with the mixed Edit and F1 results.
- Ablation Studies: JLSD plus mask-aware velocity raises MoF from 50.8 to 52.3 on HuGaDB and from 41.4 to 47.2 on LARa, with corresponding F1@50 gains.Global velocity supervision is weaker than excluding masked joints.
- Ablation Studies: MASQ remains above SMQ MoF for every tested codebook size, but the best results occur at the class-matched resolution and larger LARa codebooks decline.The sweep shows improvement is not confined to one exact quantization resolution.
Conclusion
MASQ combines full-trajectory joint masking with visible-joint temporal regularization for unsupervised skeleton action segmentation. Across HuGaDB, LARa, and BABEL, it improves frame-level action discovery, while BABEL results expose boundary jitter from hard quantization.
- MASQ is a support-aware framework for unsupervised skeleton action segmentation.It uses full-trajectory JLSD and mask-aware velocity loss to avoid artificial temporal gaps and regularize only visible joints.
- Full-trajectory JLSD removes entire joint trajectories without introducing artificial temporal gaps.The masking design supports motion inference from remaining visible body parts.
- The mask-aware velocity loss regularizes only visible joints during temporal smoothing.This separates valid temporal supervision from masked support.
- MASQ improves frame-level action discovery and often reduces fragmentation across HuGaDB, LARa, and BABEL.Ablations indicate the gains are not tied to frame masking or a single codebook size.
- Hard nearest-neighbor quantization can still cause boundary jitter near rapid transitions in BABEL.Future work targets softer assignments or explicit boundary models to improve Edit and F1 while preserving frame-level discrimination.