Source-linked AI summary
Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks
Yanghao Li, Cuiling Lan, Junliang Xing, Wenjun Zeng, Chunfeng Yuan, Jiaying Liu
TL;DR
Online action detection from streaming skeleton data must recognize action types and localize temporal boundaries before actions are fully observed. The paper proposes an end-to-end Joint Classification-Regression RNN with LSTM temporal modeling, classification, and regression, evaluated on a new annotated dataset and G3D. Experiments on both datasets demonstrate the method’s effectiveness, including more accurate localization and advance forecasting.
Problem
Online action detection must identify action types and temporal boundaries from untrimmed streaming skeleton data rather than fully observed, pre-segmented sequences.
Method
An end-to-end Joint Classification-Regression RNN uses shared deep LSTM modeling with frame-wise classification and temporal localization regression.
Results
Experiments on two datasets demonstrate effectiveness, with more accurate action-boundary localization and advance forecasting reported in the conclusion.
Takeaways & Limitations
The framework supports efficient online detection and forecasting without the typical sliding-window design.
Takeaways & Limitations
The G3D dataset is somewhat unrealistic because action number and order are fixed and actors remain motionless between actions.
Abstract
from arXiv · showhide
Human action recognition from well-segmented 3D skeleton data has been intensively studied and has been attracting an increasing attention. Online action detection goes one step further and is more challenging, which identifies the action type and localizes the action positions on the fly from the untrimmed stream data. In this paper, we study the problem of online action detection from streaming skeleton data. We propose a multi-task end-to-end Joint Classification-Regression Recurrent Neural Network to better explore the action type and temporal localization information. By employing a joint classification and regression optimization objective, this network is capable of automatically localizing the start and end points of actions more accurately. Specifically, by leveraging the merits of the deep Long Short-Term Memory (LSTM) subnetwork, the proposed model automatically captures the complex long-range temporal dynamics, which naturally avoids the typical sliding window design and thus ensures high computational efficiency. Furthermore, the subtask of regression optimization provides the ability to forecast the action prior to its occurrence. To evaluate our proposed model, we build a large streaming video dataset with annotations. Experimental results on our dataset and the public G3D dataset both demonstrate very promising performance of our scheme.
1 Introduction
Online action detection from streaming skeleton data must identify action types and localize their temporal boundaries on the fly, while ideally forecasting impending starts and ongoing ends. The paper addresses this with an end-to-end Joint Classification-Regression RNN using LSTM temporal modeling and a joint classification-regression objective.
- Problem: Online action detection identifies action types and localizes start and end points from untrimmed streaming data as early as possible.It can also forecast action starts and ends ahead of their occurrence.
- Motivation: Few works specifically address online action detection, while existing methods commonly use sliding windows for temporal localization.The paper motivates an efficient streaming system that avoids this design.
- Approach: The proposed Joint Classification-Regression RNN uses deep LSTM layers to model long-range temporal dynamics for online detection and forecasting.Its recurrent design avoids requiring a sliding window or explicit forward and backward observation.
- Approach: The network is trained end-to-end with frame-wise action classification and temporal localization regression objectives.The classification subnetwork produces the current frame’s action-class probability distribution.
- Contributions: The paper investigates online action detection from streaming skeleton data and builds a large annotated streaming action dataset for evaluation.The study also evaluates the method using the public G3D dataset.
2 Related Work
Prior action research largely focuses on recognizing pre-segmented sequences or detecting actions offline, whereas streaming detection requires timely recognition and temporal localization. The paper positions its recurrent joint classification-regression framework as an efficient alternative to sliding-window and action-proposal approaches.
- Existing action recognition and detection: Most prior methods recognize action types from well-segmented sequences or perform offline detection after the sequence is fully observed.Online recognition is motivated by applications that require reduced response delay.
- Existing localization methods: Existing streaming localization methods use sliding windows or action proposals, which can reduce computational efficiency or localization accuracy.Sliding-window size is also difficult to determine.
- Proposed direction: The proposed framework predicts action states at each time slot without a sliding window and learns start and end points through supervised regression.The regression design also supports forecasting impending starts and ongoing ends.
- Recurrent modeling: RNN-based action detection and forecasting remain less studied than action recognition, despite RNNs’ advantages for sequence learning.The paper uses LSTM memory to avoid observation-window assignment and repeated computation.
3 Problem Formulation
The formulation distinguishes offline detection, which uses the full sequence, from online detection, which must classify current frames without future information while forecasting action boundaries.
- 3.1 Offline Action Detection: Offline detection assigns each frame to one of M action classes or a blank class using the entire observed sequence.
- 3.1 Offline Action Detection: The offline objective selects the frame label with maximum posterior probability conditioned on the complete sequence.
- 3.1 Offline Action Detection: Each occurring action is represented by its class type and its starting and ending times.
- 3.2 Online Action Detection: Online detection determines the current action status using only frames observed up to the current time, without future information.
- 3.2 Online Action Detection: The online system must estimate action labels and forecast starting and ending times before their occurrence.
- 3.2 Online Action Detection: The expected forecasting interval spans [t_i,start − T, t_i,start] for starts and [t_i,end − T, t_i,end] for ends.
- 3.2 Online Action Detection: Online predictions are formulated through the joint conditional probability P(y_t, a_t, b_t|v_0, ..., v_t).
- 3.2 Online Action Detection: An LSTM neuron uses input, forget, and output gates around a memory cell that stores information over time.
4 Joint Classification-Regression RNN for Online Action Detection
The proposed framework combines a shared deep LSTM with classification and regression subnetworks to detect, localize, and forecast actions in streaming data. Joint optimization uses frame-wise action probabilities and temporal confidence peaks while avoiding explicit sliding-window lookback.
- Overview: The architecture uses a shared deep LSTM for feature extraction and temporal-dynamics modeling, followed by classification and regression subnetworks.The deep LSTM is constructed by stacking three LSTM layers and three nonlinear fully connected layers.
- Classification subnetwork: The classification subnetwork applies deep LSTM features, FC1, and SoftMax to produce the current frame’s action-class probability distribution.It is trained end-to-end with a cross-entropy objective for frame-wise action recognition.
- Joint optimization: Joint fine-tuning combines frame-wise classification with regression of the nearest action’s start and end points, enabling simultaneous recognition, localization, and forecasting.The regression loss is squared error and is weighted by λ in the overall objective.
- Boundary regression: Gaussian-like confidence targets centralize at actual action boundaries, so regressed peak positions identify start or end times.At the actual start point, the confidence equals 1; σ controls the curve shape.
- Forecasting: Confidence thresholds allow the system to forecast impending starts or ends, with higher thresholds producing later but more accurate forecasts.Frames before the confidence peak can indicate that an action will begin soon.
- Regression subnetwork: The Soft Selector performs class-specific element-wise multiplication between SoftMax outputs and FC2 features to guide regression by action class.The classification output acts as class-based feature selection for the regression subnetwork.
5 Experiments
Experiments evaluate JCR-RNN on the newly collected OAD dataset and the public G3D dataset using detection, localization, forecasting, and speed measures. The method achieves the strongest reported detection and localization performance, substantially improves forecasting, and runs at 1230 fps on skeleton input.
- Datasets and Settings: G3D contains 20 gaming actions in seven categories, but fixed action order and motionless intervals between actions make it less realistic.The paper evaluates both this existing dataset and its newly collected OAD dataset.
- Datasets and Settings: The OAD dataset contains long Kinect v2 streaming sequences with 10 freely performed actions, arbitrary order and duration, and variable idle periods.These properties are intended to approximate realistic online action detection from streaming videos.
- Evaluation Protocols: The evaluation uses F1-Score for interval detection, SL-Score for start localization, and EL-Score for end localization.F1 counts a detection as correct when predicted and ground-truth intervals overlap beyond a threshold such as 60%; SL- and EL-Scores measure relative timing accuracy.
- Action Detection Performance Evaluation: RNN-SW achieves 6% higher F1-Score than SVM-SW, while JCR-RNN improves over RNN-SW by 5.3% on the OAD dataset.The authors attribute the improvement to recurrent temporal modeling and end-to-end joint classification-regression without sliding windows.
- Action Detection Performance Evaluation: JCR-RNN achieves the best localization accuracy on OAD, and results on G3D show consistent trends across the evaluated score types.The G3D comparison also includes action-based F1, which evaluates only whether the predicted start is within 4 frames of ground truth.
- Action Forecast Performance Evaluation: At 40% recall, forecast precision is 28% for start forecasts and 37% for end forecasts, while JCR-RNN substantially outperforms the baselines.The paper explains that regression assigns informative confidence values to pre-action frames, unlike baselines that treat them as background.
- Running Speed: JCR-RNN runs at 1230 fps for skeleton-based action detection, while RNN-SW is slower because of its sliding-window design.The speed comparison uses nine long sequences averaging 3200 frames.
6 Conclusion and Future Work
The paper concludes that its Joint Classification-Regression RNN detects and localizes actions online while forecasting their occurrence in advance. Future work will add appearance and human-object interaction features to improve detection and forecast performance.
- The Joint Classification-Regression RNN localizes action start and end points on the fly while determining action types.
- Deep LSTM modeling captures complex long-range temporal dynamics without the typical sliding window design.
- The regression component improves localization accuracy and forecasts action occurrence in advance.
- Experiments on two datasets demonstrate the effectiveness of the proposed method.
- Future work will incorporate appearance and human-object interaction information to further improve detection and forecast performance.
1 Experimental Results on the OAD Dataset
On the OAD dataset, the study compares the proposed method with and without the Soft Selector Module and with sliding-window baselines across detection and localization measures.
- The Soft Selector Module produces significant improvement on the OAD dataset compared with directly linking the FC2 and FC3 layers.The comparison uses the same parameters in both settings and reports F1-, SL-, and EL-Scores.
- The OAD evaluation reports F1-Score for the proposed method and sliding-window baselines using different window sizes.
- SL- and EL-Scores are reported for the sliding-window baselines across different window sizes.The window stride is set to half of the window size.
2 Experimental Results on the G3D Dataset
On the G3D dataset, the study evaluates all seven video categories using frame-level detection, start/end localization, and action-based F1 measures.
- Results are reported for all seven G3D video categories, extending beyond the first two categories shown in the paper because of space limitations.
- The G3D evaluation uses F1-Score and SL- and EL-Scores to assess detection and temporal localization.
- Action-based F1 is used as an additional comparison metric on the G3D dataset.
- The reported G3D results are summarized in tables for F1-Score, SL- and EL-Score, and action-based F1.