PaperlayerView exact revision on arXiv

Source-linked AI summary

Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention

Yuxiao Chen, Long Zhao, Xi Peng, Jianbo Yuan, Dimitris N. Metaxas

arXiv:1907.08871v1cs.CV

TL;DR

Deep learning methods for hand gestures do not explicitly model skeleton structure and temporal dynamics, and predefined graphs may not adapt to action-specific variation. DG-STA learns graph structure and node features with spatial-temporal attention, and the paper reports state-of-the-art performance on two benchmarks.

  • Problem

    Deep learning methods do not explicitly exploit hand-skeleton structure and temporal dynamics, while fixed graph structures lack flexibility across actions.

    Source: §1 · p. 2

  • Method

    DG-STA dynamically learns hand-skeleton graph edges and node features with spatial-temporal attention, using position embeddings and a spatial-temporal mask.

    Source: §1 · pp. 2–3

  • Results

    DG-STA achieves state-of-the-art performance on the DHG-14/28 and SHREC’17 benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The framework is presented as applicable to other graph-based tasks that learn spatial and temporal information, such as skeleton-based human action recognition.

    Source: §5 · p. 10

Abstract

from arXiv · show

We propose a Dynamic Graph-Based Spatial-Temporal Attention (DG-STA) method for hand gesture recognition. The key idea is to first construct a fully-connected graph from a hand skeleton, where the node features and edges are then automatically learned via a self-attention mechanism that performs in both spatial and temporal domains. We further propose to leverage the spatial-temporal cues of joint positions to guarantee robust recognition in challenging conditions. In addition, a novel spatial-temporal mask is applied to significantly cut down the computational cost by 99%. We carry out extensive experiments on benchmarks (DHG-14/28 and SHREC'17) and prove the superior performance of our method compared with the state-of-the-art methods. The source code can be found at https://github.com/yuxiaochen1103/DG-STA.

1 Introduction

Skeleton-based gesture recognition uses joint sequences, but existing deep methods do not explicitly model skeleton structure and temporal dynamics, while fixed graphs cannot adapt to different actions. DG-STA learns an action-specific graph through spatial-temporal attention and adds position embeddings and a mask operation.

  • Skeleton-based methods use sequences of hand-joint coordinates and can be more robust to lighting variation and occlusion when joint coordinates are accurate.Source: §1 · p. 1
  • Deep learning methods improve on hand-crafted descriptors, but they do not explicitly exploit hand-skeleton spatial structure and temporal dynamics.Source: §1 · p. 2
  • Predefined skeleton graphs lack flexibility to represent action-specific structural and dynamic variation, which can yield suboptimal performance.Source: §1 · p. 2
  • DG-STA dynamically optimizes graph edges and nodes with self-attention in spatial and temporal domains to model different actions.Source: §1 · pp. 2–3
  • DG-STA outperforms state-of-the-art methods on the DHG-14/28 and SHREC’17 benchmarks.Source: §1 · pp. 2–3
  • Spatial-temporal position embeddings encode node identity and temporal order, while a spatial-temporal mask supports efficient implementation.Source: §1 · p. 3

2 Related Work

Related work covers hand-gesture descriptors and deep models, self-attention for graph representations, and skeleton graphs for action recognition. The paper distinguishes its approach by explicitly representing temporal information and learning action-specific graph structures.

  • Unlike sequence self-attention and GAT node embeddings, the paper's approach captures temporal information and node identities in graph-based hand skeletons.Source: §2 · p. 3
  • Hand-gesture methods progress from handcrafted descriptors to deep networks, but learning-based methods do not explicitly explore hand structure and dynamics.Source: §2 · p. 3
  • Skeleton-action graphs often use predefined body connections, whereas DG-STA uses multi-head attention to learn multiple action-specific graphs.Source: §2 · p. 3

3 Methodology

DG-STA builds a fully connected hand-skeleton graph and uses spatial and temporal attention to learn its node features and edge weights. Spatial-temporal position embeddings add joint identity and time-step information, while masks make the implementation more efficient.

3.1 Skeleton Graph Initialization

  • Each video frame contributes N hand-joint nodes, and every node connects to all nodes, including itself, through spatial, temporal, or self-connected edges.Source: §3.1 · p. 4

3.2 Dynamic Graph Construction via Spatial-Temporal Attention

  • DG-STA applies spatial attention followed by temporal attention, then average-pools node features into a sequence representation for gesture classification.Spatial attention learns weighted graph structures; temporal attention adds information across time steps.Source: §3.2 · pp. 4–5
  • Each spatial-attention head computes scaled dot-product weights among nodes at the same time step and uses them to aggregate value vectors.Temporal edges receive zero weight in the spatial attention model.Source: §3.2 · pp. 4–5

3.3 Spatial-Temporal Position Embedding

  • Spatial-temporal position embeddings add joint identity and time-step information to node features before DG-STA processes them.The spatial embedding has N joint vectors; the temporal embedding has N × T node vectors, with sine and cosine values at different frequencies.Source: §3.3 · pp. 5–6

3.4 Efficient Implementation

  • The spatial-temporal mask operation reduces computation time by 99%.It masks scaled dot-products so attention focuses on spatial or temporal edges; the spatial mask preserves spatial and self-connected edges.Source: §3.4 · pp. 6–7, Figure 2 · p. 6

4 Experiments

The experiments evaluate DG-STA on two hand-gesture benchmarks, test its components, and compare it with prior methods. The reported results favor the full model across the evaluated settings, with one comparable result on SHREC’17.

4.1 Implementation Details

  • The network adds spatial position embeddings before spatial attention, then temporal position embeddings before temporal attention, and average-pools node features for classification.The spatial and temporal attention models each use 8 heads; query, key, and value dimensions are 32.Source: §4.1 · p. 7

4.2 Datasets and Settings

  • The benchmarks contain 2,800 sequences of 14 gestures in single-finger or whole-hand configurations, with 22 three-dimensional hand joints provided per frame.DHG-14/28 uses leave-one-subject-out evaluation across 20 folds; SHREC’17 uses its provided split, with accuracy reported for 14 and 28 gestures.Source: §4.2 · pp. 7–8

4.3 Ablation Study

  • The full DG-STA achieves the best performance in the ablation study across DHG-14/28 and SHREC’17.The study evaluates the fully-connected graph structure, spatial-temporal attention, and spatial-temporal position embedding.Source: Table 1 · p. 8, §4.3 · p. 8
  • The fully-connected graph model significantly outperforms the sparse skeleton graph model, which may be suboptimal for some gestures.The authors attribute this to the fully-connected graph's greater flexibility to learn action-specific graph structures.Source: §4.3 · p. 8
  • The spatial-temporal attention model performs better than the variant using one attention model over the whole graph.The variant replaces separate spatial and temporal attention models with a single graph-attention model.Source: §4.3 · p. 8
  • DG-STA outperforms the variant without spatial-temporal position embedding, supporting the value of encoded node identity and temporal order information.Source: §4.3 · pp. 8–9

5 Conclusions

The conclusion presents DG-STA as a graph-based spatial-temporal attention method for skeleton-based hand-gesture recognition. It reports experimental effectiveness and identifies graph-based spatial-temporal learning as a potential framework for other tasks.

5 Conclusions

  • DG-STA uses spatial and temporal attention on a fully-connected hand-skeleton graph to learn edge weights and extract spatial and temporal information.Source: §5 · p. 10
  • The authors report that extensive experiments demonstrate the effectiveness of the framework.Source: §5 · p. 10
  • The authors identify skeleton-based human action recognition as a possible further use of the framework for learning spatial and temporal information from graph-based data.Source: §5 · p. 10
Loading 1907.08871v1…