Source-linked AI summary
Interpretable 3D Human Action Analysis with Temporal Convolutional Networks
Tae Soo Kim, Austin Reiter
TL;DR
3D skeleton-based action recognition is effective, but the temporal models used for it are largely black-box. The paper redesigns temporal convolutional networks with residual connections for interpretable spatio-temporal representations, and Res-TCN achieves state-of-the-art results on NTU-RGBD.
Problem
Learning-based temporal models for 3D human action recognition remain mostly black-box despite interpretable 3D skeleton inputs.
Method
The paper applies TCNs to 3D skeleton sequences and redesigns deeper layers as additive residual terms for interpretable representations and parameters.
Results
Res-TCN achieves state-of-the-art results on the NTU-RGBD dataset and produces explainable, discriminative spatio-temporal features.
Takeaways & Limitations
Res-TCN makes learned filters, features, and spatial-temporal attention more directly interpretable while improving upon state-of-the-art performance.
Takeaways & Limitations
The model interpretation focuses on selected hidden-representation dimensions with significant weight magnitudes; other positive dimensions also contribute to classification.
Abstract
from arXiv · showhide
The discriminative power of modern deep learning models for 3D human action recognition is growing ever so potent. In conjunction with the recent resurgence of 3D human action representation with 3D skeletons, the quality and the pace of recent progress have been significant. However, the inner workings of state-of-the-art learning based methods in 3D human action recognition still remain mostly black-box. In this work, we propose to use a new class of models known as Temporal Convolutional Neural Networks (TCN) for 3D human action recognition. Compared to popular LSTM-based Recurrent Neural Network models, given interpretable input such as 3D skeletons, TCN provides us a way to explicitly learn readily interpretable spatio-temporal representations for 3D human action recognition. We provide our strategy in re-designing the TCN with interpretability in mind and how such characteristics of the model is leveraged to construct a powerful 3D activity recognition method. Through this work, we wish to take a step towards a spatio-temporal model that is easier to understand, explain and interpret. The resulting model, Res-TCN, achieves state-of-the-art results on the largest 3D human action recognition dataset, NTU-RGBD.
1. Introduction
3D skeleton sequences preserve spatial and temporal information for human activity recognition, but learning-based temporal models remain difficult to interpret. The paper proposes an interpretable Res-TCN approach and reports state-of-the-art NTU-RGBD performance.
- Motivation: RGB-video methods project 3D human motion into 2D images, inevitably losing valuable 3D spatio-temporal information.
- Motivation: 3D skeletons represent human motion as interpretable 3D joint trajectories and provide a discriminative, robust representation for activity analysis.Each feature dimension can correspond to a joint coordinate over time.
- Motivation: Despite progress in 3D skeleton-based recognition, the inner workings of complex temporal models remain mostly black-box.
- Approach: The paper applies Temporal Convolutional Neural Networks to 3D human action recognition to examine what models learn from interpretable skeleton sequences.
- Approach: Res-TCN factors deeper layers into additive residual terms, producing interpretable hidden representations and model parameters.
- Results: Res-TCN obtains state-of-the-art results on the NTU-RGBD 3D human activity recognition dataset.
2. Related Work
Prior work largely uses LSTM-based recurrent models for 3D human action recognition, while interpretability remains difficult. The paper positions Res-TCN as a directly interpretable alternative using residual temporal convolutions.
- 3D Human Activity Recognition: Most leading 3D human action recognition methods adopt LSTM-based RNNs because their recurrent structure models temporal dependencies.
- 3D Human Activity Recognition: Prior models organize joints hierarchically, by spatial groups, or through co-occurrence constraints to capture discriminative body-part relationships.
- Interpretability Gap: Although prior methods emphasize important joint subsets, interpreting the parameters of individual LSTM layers remains difficult.
- Interpretability Gap: Res-TCN learns spatial and temporal attention without the initial pre-training stage used by a cited LSTM attention model.
- Interpretability Gap: Residual connections and temporal convolutions let the proposed architecture directly interpret its parameters and features.
- Model Interpretability: Post-hoc explanations use stimuli, examples, or prototypes, but the causal relationship between model parameters and predictions remains unclear.
- Model Interpretability: The paper focuses on interpreting TCN representations and designing readily interpretable hidden representations and parameters for 3D action analysis.
3. Overview of Temporal Convolutional Neural Networks
A TCN applies stacked one-dimensional temporal convolutions and nonlinearities to frame-wise video features. The paper adapts its encoder for action recognition and adds residual connections to improve interpretability.
- TCN Structure: The original TCN uses a convolutional encoder-decoder for temporal action segmentation; this work adapts its encoder for action recognition.
- TCN Structure: A TCN stacks units of one-dimensional convolution and nonlinear activation, with convolution performed across the temporal domain.
- TCN Input: For T video frames with D-dimensional features, the input is a concatenation of frame-wise features X ∈ R^T × D.
- TCN Structure: Repeated convolution-and-activation blocks extract features, with each layer using multiple filters over a temporal window.
- Residual Design: The model uses residual connections, and except for the first convolution layer, its architecture consists of stacked residual units.
- Residual Design: Residual connections are adopted to further improve TCN interpretability, while the network is trained with back-propagation.
4. Interpretability of TCNs with Residual Connections
Res-TCN uses residual temporal convolutions to keep hidden representations connected to interpretable skeleton features. This design enables tracing deeper filters back to first-layer filters and then to specific joints and motion patterns.
- Input Representation: Frame-wise skeleton features provide interpretable input dimensions, allowing first-layer filters and activation maps to retain semantic meaning.Each feature dimension corresponds to a spatial configuration, such as a joint’s X, Y, or Z coordinate.
- Residual Architecture: Res-TCN uses residual units with pre-activation ReLU, adding transformed positive-region signals to the preceding representation.The first convolution operates directly on raw skeleton input, while later units compute residual updates before merging them with their inputs.
- Residual Architecture: Because each residual update is added to X1, deeper representations can be interpreted as additions or subtractions in the common language of first-layer features.This formulation forces discriminative spatio-temporal features to remain connected to the directly interpretable first-layer representation.
- First-Layer Filters: A first-layer filter jointly examines multiple time steps across all feature dimensions and can directly encode the motion of selected joints.The example filter has parameters near zero except for dimensions associated with joints 11 and 25.
- Deeper-Layer Filters: In deeper layers, filters act as gates that determine how much transformed information from lower-layer features is added to the residual output.Large weights emphasize particular dimensions, while low-magnitude weights contribute less; influential dimensions can be recursively traced to first-layer filters and skeleton motion.
5. Experiments
Experiments evaluate Res-TCN on the NTU RGB+D skeleton dataset and use its hidden representations and filters to produce interpretable action explanations. The model’s activations describe coordinated joint movements over time while retaining discriminative performance, including improvements over prior methods in both evaluation settings.
- Dataset and Settings: NTU RGB+D contains 56,880 training videos spanning 60 action classes, with Cross-Subject and Cross-View evaluation settings.The dataset includes full 3D skeleton annotations, 40 subjects, and three camera viewpoints.
- Why Did My Model Predict This?: Res-TCN explanations use model parameters and hidden representations to address why a prediction was made.The approach analyzes selected hidden dimensions and their corresponding filters rather than relying only on post-hoc examples.
- Why Did My Model Predict This?: The X4 activation map identifies temporally ordered responses for left ankle and hip translation, right-knee movement, and left-wrist/right-ankle swinging.The green, yellow, and blue filters correspond to these distinct joint-related responses over the sequence.
- Why Did My Model Predict This?: Filter activations align with the kicking sequence: left ankle and hip responses approach zero as the right-knee response peaks.The activation changes provide a time-localized account of the pivot-foot, kicking-foot, and return motions.
- Why Did My Model Predict This?: The interpretation focuses on selected hidden dimensions with significant weight magnitudes, while other positive dimensions also contribute to the classifier’s final decision.Thus, the illustrated explanation is selective rather than a complete account of every contributing dimension.
- Comparison to Other State-of-the-Art: Res-TCN improves on the current state of the art on NTURGB+D in both Cross-Subject and Cross-View accuracy settings.The cited comparison reports performance against published learning-based methods, while the table caption identifies accuracy as the metric.
6. Conclusion
The conclusion presents Res-TCN as an interpretable approach to 3D human action analysis using skeleton sequences. It reports that learned filters capture different spatial and temporal attention patterns while the model produces discriminative representations and improves upon the state of the art.
- Conclusion: Res-TCN is presented as a new approach for 3D human action analysis that enhances interpretability of model parameters and features.The conclusion contrasts its interpretability with popular RNN-based approaches.
- Conclusion: Skeleton positions let the model’s learned filters be examined for the spatial and temporal information they use in making predictions.The paper reports different levels of attention across both spatial and temporal dimensions.
- Conclusion: Res-TCN is reported to be explainable, discriminative, and improved over the state of the art for human activity analysis.These properties are presented as experimentally validated conclusions.