PaperlayerView exact revision on arXiv

Source-linked AI summary

Feedback Graph Convolutional Network for Skeleton-based Action Recognition

Hao Yang, Dan Yan, Li Zhang, Dong Li, YunDa Sun, ShaoDi You, Stephen J. Maybank

arXiv:2003.07564v1cs.CV

TL;DR

Conventional feedforward GCNs make it difficult for low-level layers to access high-level semantic information when modeling skeleton actions. FGCN combines multi-stage temporal sampling with feedback connections and early predictions, and the paper reports state-of-the-art performance on three action-recognition datasets.

  • Problem

    Conventional feedforward GCNs cannot provide low-level layers access to high-level semantic information, while useful action cues may be buried in entire-sequence inputs.

    Source: §1 · p. 2

  • Method

    FGCN sparsely samples skeleton clips across temporal stages and uses a dense feedback block to pass high-level semantics and prior-stage information through the network.

    Source: §1 · p. 2, §3.3 · p. 8

  • Results

    FGCN achieves state-of-the-art performance on NTU-RGB+D, NTU-RGB+D120, and Northwestern-UCLA.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    Early predictions from partial skeleton sequences can guide later feature learning and support timely decisions in latency-sensitive applications such as robotics or autonomous driving.

    Source: §1 · p. 3

Abstract

from arXiv · show

Skeleton-based action recognition has attracted considerable attention in computer vision since skeleton data is more robust to the dynamic circumstance and complicated background than other modalities. Recently, many researchers have used the Graph Convolutional Network (GCN) to model spatial-temporal features of skeleton sequences by an end-to-end optimization. However, conventional GCNs are feedforward networks which are impossible for low-level layers to access semantic information in the high-level layers. In this paper, we propose a novel network, named Feedback Graph Convolutional Network (FGCN). This is the first work that introduces the feedback mechanism into GCNs and action recognition. Compared with conventional GCNs, FGCN has the following advantages: (1) a multi-stage temporal sampling strategy is designed to extract spatial-temporal features for action recognition in a coarse-to-fine progressive process; (2) A dense connections based Feedback Graph Convolutional Block (FGCB) is proposed to introduce feedback connections into the GCNs. It transmits the high-level semantic features to the low-level layers and flows temporal information stage by stage to progressively model global spatial-temporal features for action recognition; (3) The FGCN model provides early predictions. In the early stages, the model receives partial information about actions. Naturally, its predictions are relatively coarse. The coarse predictions are treated as the prior to guide the feature learning of later stages for a accurate prediction. Extensive experiments on the datasets, NTU-RGB+D, NTU-RGB+D120 and Northwestern-UCLA, demonstrate that the proposed FGCN is effective for action recognition. It achieves the state-of-the-art performance on the three datasets.

1 Introduction

The introduction motivates FGCN as a coarse-to-fine skeleton action recognizer that addresses information-flow limits in feedforward GCNs. It presents feedback connections and early predictions as central contributions, and reports state-of-the-art performance across three datasets.

1 Introduction

  • Conventional feedforward GCNs process entire skeleton sequences, where useful action information may be buried in irrelevant clips, and low-level layers cannot access high-level semantics.The introduction also notes that processing the entire sequence increases computational complexity.Source: §1 · p. 2
  • FGCN sparsely samples clips across temporal stages and uses a dense Feedback Graph Convolutional Block to pass high-level semantics to low-level layers and carry information between stages.The block's lateral connections support progressive modeling of global spatial-temporal features.Source: §1 · pp. 2–3
  • FGCN produces coarse early predictions from partial action information, then uses them as priors to guide later-stage feature learning toward more accurate predictions.The introduction identifies robotics and autonomous driving as settings where early predictions may be valuable because latency is crucial.Source: §1 · p. 3
  • FGCN achieves state-of-the-art performance on NTU-RGB+D, NTU-RGB+D120, and Northwestern-UCLA.The paper reports extensive evaluation on all three datasets.Source: §1 · p. 3

2 Related Works

The related work reviews skeleton action-recognition representations, graph-based methods, and feedback mechanisms in vision. It situates FGCN against limitations in spatial-joint modeling and feedforward information flow.

2.1 Skeleton based Action Recognition

  • Earlier skeleton methods use coordinate-vector sequences with RNNs or pseudo-images with CNNs, but do not explicitly model dependencies between correlated joints [27].Skeleton data is described as robust to illumination change, scene variation, and complex backgrounds.Source: §2.1 · pp. 3–4

2.2 GCN based Action Recognition

  • GCN-based methods model skeletons as spatial-temporal graphs, but conventional feedforward GCNs prevent low-level layers from accessing high-level semantic information.The reviewed GCN approaches include spatial and spectral graph-convolution perspectives.Source: §2.2 · p. 4

2.3 Feedback Network

  • Feedback mechanisms have been used in object recognition, super resolution, and pose estimation, including transferring high-level information to refine lower-level representations.The review describes feedback as carrying output information to correct previous states.Source: §2.3 · pp. 4–5

3 The Method

FGCN samples skeleton sequences across temporal stages, then uses feedback and dense connections to progressively combine local features into global spatial-temporal representations. It predicts at each stage and supports a two-stream spatial-and-motion framework.

3.1 Graph Convolutional Network

  • The skeleton is represented as a sequence of graphs, with joints as vertices, bones as edges, and graph convolution operating on each vertex and its neighbors.The neighbor set uses 1-distance neighbors, and the experiments divide it into three subsets.Source: §3.1 · p. 5

3.2 Feedback Graph Convolutional Network

  • FGCN samples clips from equal-interval temporal stages, extracts local features, and passes them through FGCB to combine information across stages.Each stage predicts an action; last-win-all, average, and weighting fusion strategies combine the local predictions into a video-level prediction.Source: §3.2 · pp. 6–7

3.3 Feedback Graph Convolutional Block

  • FGCB uses dense shortcut connections so each layer receives preceding-layer features, while prior-stage high-level information modulates the current stage's low-level input.Its final layer compresses and fuses feature maps from preceding layers to produce the block output.Source: §3.3 · p. 8

3.4 Two-stream Framework of FGCN

  • The two-stream framework feeds spatial skeleton graphs and temporal motion graphs into separate FGCN models, then weights and fuses their softmax output scores.Joint and bone motion is defined as the difference between their vectors in consecutive frames.Source: §3.4 · p. 9

4 Experiments

Experiments evaluate FGCN on three skeleton-action datasets and compare its results with prior methods. Ablations examine temporal sampling, fusion strategies, and input modalities.

4.1 Datasets

  • Evaluation covers NTU-RGB+D (60 classes; cross-subject/view), NTU-RGB+D120 (120 classes; cross-subject/setup), and Northwestern-UCLA (10 actions; camera-based train/test).Source: §4.1 · pp. 9–10

4.3 Ablation Study

  • Six stages performed best, but five stages with 64 frames per stage balanced performance and computational cost; average fusion performed best among three temporal strategies.The study evaluated stage count and per-stage frame length, and compared last-win-all, average, and weighting fusion.Source: Figure 4 · p. 10, Table 1 · p. 11, §4.3 · pp. 11–12

4.2 Implementation Details

  • Experiments sampled 64 frames from each of five temporal stages, using ten front-end graph-convolution layers and a four-layer FGCB.The reported setup uses spatial-temporal kernel sizes of 3 and 256 output channels in the FGCB.Source: §4.2 · pp. 10–11

4.3 Ablation Study

  • FGCN-joint outperformed same-architecture ST-GCN-joint by 5.54% cross-subject and 5.27% cross-view; joint+bone fusion also improved over either model.The authors attribute the joint-model gains to introducing feedback into GCNs.Source: §4.3 · p. 12
  • Spatial+motion fusion reached 90.22% cross-subject and 96.25% cross-view, improving on both inputs; spatial-only reached 88.32% and 94.82%, and motion-only 85.96% and 93.57%.Source: §4.3 · p. 12

4.4 Comparison with State-of-the-art

  • FGCN is state-of-the-art on all three datasets, beating ST-GCN by 8.7% cross-subject/8.0% cross-view on NTU-RGB+D and 13.0% cross-subject/16.1% cross-setup on NTU-RGB+D120.The paper also reports state-of-the-art performance on Northwestern-UCLA.Source: Table 3 · p. 13, Table 4 · p. 13, §4.4 · pp. 13–14

5 Conclusion

FGCN extracts skeleton action features through multi-stage temporal sampling and feedback connections between graph-convolutional layers and stages. It provides early predictions and achieves state-of-the-art performance on three action-recognition datasets.

5 Conclusion

  • FGCN sparsely samples skeleton clips across temporal stages and uses graph-convolutional layers to extract local spatial-temporal features.Its Feedback Graph Convolutional Block sends semantic information from high-level layers to low-level layers and carries information across stages.Source: §5 · p. 14
  • FGCN achieves state-of-the-art action-recognition performance on NTU-RGB+D, NTU-RGB+D120, and Northwestern-UCLA.The model is extensively evaluated on all three datasets.Source: §5 · p. 15
  • FGCN provides early predictions during its progressive process.These predictions help agents make timely decisions on-the-fly.Source: §5 · p. 15

Related papers

Loading 2003.07564v1…