PaperlayerView exact revision on arXiv

Source-linked AI summary

De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift

Mengyuan Liu, Yuhang Wen, Yi Zhang, Songtao Wu, Hong Liu, Junsong Yuan, Beichen Ding

arXiv:2609.32454v1cs.CVcs.AIcs.LG

TL;DR

Late-fusion recognition assumes entities are independently and identically distributed, but fixed coordinate origins can create entity-distribution discrepancies in skeleton data. CHASE adaptively shifts skeletons within their convex hull and uses distribution alignment, with experiments reporting consistent improvements across seven datasets and varied recognition tasks.

  • Problem

    Late-fusion skeleton recognition assumes entities are independently and identically distributed, while fixed coordinate origins can create inter-entity distribution discrepancies that violate this assumption.

    Source: §1 · pp. 2–3

  • Method

    CHASE normalizes skeletons with sample-adaptive shifts constrained to the convex hull, a pairwise distribution-distance objective, and a sub-entity strategy for single-entity settings.

    Source: §1 · p. 3, §3.2 · p. 8, §3.4 · p. 10

  • Results

    CHASE consistently improves recognition across seven datasets and diverse interaction types and backbone architectures.

    Source: §1 · p. 3, §5 · p. 18

  • Takeaways & Limitations

    As a plug-and-play normalization module, CHASE supports improved recognition across diverse settings and multiple skeletal modalities.

    Source: §5 · p. 18

    Coordinate normalization addresses the removable origin-offset portion of sub-entity gaps, not the intrinsic structural differences among body parts.

    Source: §4.5 · p. 17

Abstract

from arXiv · show

Skeleton sequences can represent both individual actions and multi-entity interactions, encompassing human bodies, hands, objects, and robots. Existing approaches to recognize skeleton-based actions and interactions usually adopt a late fusion strategy, which expects individuals are independent and identically distributed to train a robust weight-shared entity encoder. However, observed entity bias in various skeletal data violates this assumption, leading to suboptimal optimization of backbone models that might produce wrong recognition results. This bias arises from the world coordinate system's initial configuration, where the choice of origin often creates bias in representation. To this end, we propose a Convex Hull Adaptive Shift based normalization method to reduce Entity bias (CHASE), improving performance across a variety of skeleton-based action and interaction recognition tasks. To adaptively apply plausible shifts to the input skeletons, we formulate a plug-and-play parameterized network that ensures the relocated world origin lies within the skeleton convex hull, which avoids non-convergence by limiting the search space. To further minimize entity bias, we incorporate an auxiliary objective that leverages pair-wise distribution distances to guide network optimization. To support both single- and multi-entity actions, we propose a sub-entity strategy that offers a consistent formulation for both scenarios. Moreover, CHASE demonstrates compatibility with various intra-skeleton modalities, such as bones and velocities, highlighting its adaptability. Essentially, our method works as a normalization approach to reduce entity bias, enabling subsequent classifiers to achieve improved recognition performance across diverse settings. Extensive experiments on 7 datasets verify our approach by seamlessly integrating with various backbones and significantly boosting their performance.

1 Introduction

Late-fusion skeleton models rely on entities being independently and identically distributed, but fixed coordinate origins create distributional asymmetries that undermine this assumption. CHASE adaptively shifts skeletons within their convex hull and adds distribution alignment, extending the method to single-entity actions and diverse modalities; experiments report improvements across seven datasets.

1 Introduction

  • Late fusion assumes entities are independently and identically distributed, but skeleton data frequently violates this assumption.Fixed coordinate origins can center the first entity while spatially displacing others, producing differences in entity distributions and undermining shared-backbone optimization.Source: §1 · pp. 2–3
  • CHASE adaptively shifts skeleton sequences as a normalization step to reduce entity bias across action and interaction recognition tasks.Its learnable module constrains the relocated origin to the skeleton convex hull, while an auxiliary pairwise-distance objective guides distribution alignment.Source: §1 · p. 3
  • CHASE improves single-entity backbones to state-of-the-art performance across recognition tasks, with consistent gains across seven datasets.The paper also reports extensions to newer backbones and scenarios, including human-robot interaction.Source: §1 · pp. 3–4
  • The sub-entity strategy extends CHASE to single-entity sequences by treating one entity as structurally comparable component groups.Its criteria include subgraph and isomorphism conditions, with heuristic identification such as using physical topology.Source: §1 · p. 3
  • CHASE extends beyond joint inputs to bones, velocities, and JBF representations, with the network, objective, and sub-entity strategy reformulated for these modalities.The paper reports broader evaluations with newer backbones and scenarios, including HARPER human-robot interactions.Source: §1 · p. 4

2 Related Work

The related work distinguishes input-space entity bias from cross-domain distribution shifts and feature normalization. It surveys skeleton recognition approaches and interaction-specific models, positioning CHASE as a way to apply single-entity models across varied settings with less reliance on domain-specific priors.

2.1 Distribution Alignment and Normalization

  • Entity bias is a within-dataset asymmetry in input coordinates, unlike cross-domain shifts or feature-level normalization; CHASE learns constrained spatial shifts guided by within-batch entity alignment.Domain adaptation methods require explicit domain partitions, while standard feature normalization does not address the input geometry.Source: §2.1 · p. 4

2.2 Skeleton-based Action Recognition

  • Skeleton recognition research spans recurrent and graph-convolution models, attention methods, multiple skeletal representations, and auxiliary objectives.Late-fusion adaptation relies on an IID assumption for entities, which CHASE is proposed to address.Source: §2.2 · pp. 4–5

2.3 Skeleton-based Multi-Entity Action Recognition

  • Interaction-specific methods often use domain priors or intricate architectures, whereas CHASE supports simpler single-entity models across diverse scenarios with minimal reliance on such priors.The surveyed settings include human-human, hand-object, human-robot, and group activities; the reviewed approaches are often specialized to particular scenarios.Source: §2.3 · pp. 5–6

3 CHASE

CHASE normalizes skeleton sequences with sample-adaptive shifts and pairwise distribution alignment, then extends this framework to single-entity actions and multiple skeletal modalities.

3 CHASE

  • CHASE combines a learnable normalization module, a pairwise auxiliary objective, and extensions for single- and multi-entity actions and varied skeletal modalities.The framework is intended to reduce entity bias before a subsequent backbone processes the skeletons.Source: §3 · p. 6, Figure 2 · p. 7

3.1 Preliminaries

  • Late fusion independently encodes each entity with a weight-shared backbone and averages features, relying on an independent-and-identically-distributed assumption.Skeleton sequences are represented across coordinates, time, joints, and entities.Source: §3.1 · pp. 6–7

3.2 Convex Hull Adaptive Shift

  • CHASE learns a sample-adaptive shift and constrains the relocated origin to the skeleton convex hull, limiting the search space to avoid non-convergence.The shift is applied across the sequence points, and the coefficient mapping makes it input-dependent.Source: §3.2 · p. 8

3.3 Objective for Entity Bias Minimization

  • The auxiliary objective minimizes pairwise entity-distribution discrepancies to guide shifts away from trivial solutions and toward a shared coordinate space.It estimates expectations on mini-batches and uniformly samples entity pairs to make the objective tractable.Source: §3.3 · p. 9

3.4 Any-Entity Generalization via Sub-Entity Strategy

  • For single-entity actions, the sub-entity strategy forms structurally isomorphic body-part units so the pairwise distribution objective remains applicable.The strategy supports a unified formulation across single- and multi-entity actions.Source: §3.4 · p. 10

3.5 A Modality-Agnostic De-biasing Framework

  • CHASE extends beyond joints to point-set skeletal modalities; for bones, it selects sub-entities using subgraphs of the line graph of the original topology.The described modalities include bones, velocities, and JBF.Source: §3.5 · p. 11

4 Experiments

Experiments evaluate CHASE across diverse skeleton-based recognition benchmarks, backbones, and input modalities. CHASE improves baseline performance and reduces entity-distribution discrepancies, with ablations examining its design and scope.

4.1 Datasets & Evaluation

  • The evaluation uses a variety of skeletal datasets spanning human actions and interactions, hand-object and bi-hand interactions, human-robot interactions, and group activities, with dataset details listed in Table 1.Source: §4.1 · p. 11

4.3 Quantitative Results

  • CHASE consistently improves five single-entity backbones and can let simpler late-fusion models surpass specialized multi-entity encoders across seven datasets.The comparisons include ISTA-Net, me-GCN, ASEA, and HandFormer.Source: Table 2 · p. 12, §4.3 · pp. 13–14

4.2 Implementation Details

  • Experiments compare GCN-, hyper-GCN-, and transformer-based backbones, with benchmark-specific configurations and standard NTU26 X-Sub training details.NTU26 training uses SGD with Nesterov momentum, label smoothing, batch size 64, and 110 epochs.Source: §4.2 · p. 13

4.3 Quantitative Results

  • CHASE achieves the highest accuracy improvements among tested alternatives, while reducing inter-entity distribution discrepancies across all five reported metrics.Learnable static offsets fall short; the authors attribute CHASE’s gains to sample-adaptive shifts and the convex-hull constraint.Source: Table 3 · p. 14, §4.3 · pp. 14–15, Table 4 · p. 15

4.4 Qualitative Results

  • CHASE aligns entity distributions in mean and covariance, while its enhanced backbone learns more distinctive representations and improves accuracy for most categories.Reported category gains include wield knife (+8.50%) and shoot with gun (+8.00%).Source: Figure 6 · p. 14, §4.4 · p. 15

4.5 Ablation Study

  • Ablations show that the convex-hull constraint is pivotal to adaptive-shift learning, and the MPMMD trade-off performs best at λ = 0.1.Removing the constraint reduces accuracy by over 60% at initial learning rates of 0.1 and 0.01.Source: §4.5 · pp. 16, 18
  • For single-entity actions, defining sub-entities such as two body parts improves accuracy and model efficiency, but gains are smaller than for multi-entity actions.The authors attribute the smaller gains to less distributional discrepancy among parts of one person; anatomical heterogeneity remains after coordinate normalization.Source: §4.5 · pp. 16–17
  • CHASE improves DeGCN across joints, bones, joint velocities, and JBF, with gains varying by modality.It adds around 2.50M FLOPs; λ = 0.1 gives the best MPMMD performance, and CTR-GCN with CHASE performs better under tested noise and masking.Source: §4.5 · pp. 17–18

5 Conclusion

CHASE is a plug-and-play normalization method that adaptively shifts the world origin to reduce entity bias in skeleton-based action and interaction recognition. Experiments across five baseline backbones and seven datasets report state-of-the-art performance across varied tasks.

5 Conclusion

  • CHASE adaptively re-centers the world origin as a plug-and-play normalization module to reduce entity bias and improve subsequent classifiers.The paper presents it for skeleton-based action and interaction recognition.Source: §5 · p. 18
  • Experiments evaluate CHASE with five baseline backbones on seven datasets across varied numbers and types of entities.Source: §5 · p. 18
  • The reported experiments indicate that CHASE reaches state-of-the-art performance across a variety of action and interaction recognition tasks.Source: §5 · p. 18

Declarations

The declarations report funding, no relevant competing interests, and links to the datasets used in the paper.

5.1 Funding

  • The work lists support from several research funds and laboratories.The listed support includes the National Natural Science Foundation of China and Guangdong Outstanding Youth Fund.Source: §5.1 · p. 19

5.2 Competing Interests

  • The authors declare no competing interests relevant to the paper.Source: §5.2 · p. 19

5.3 Data Availability

  • The authors provide access links for six datasets used in the paper.The listed datasets include NTU RGB+D, H2O, Assembly101, Collective Activity, Volleyball, and HARPER.Source: §5.3 · p. 19

Related papers

Loading 2609.32454v1…