Source-linked AI summary
Temporal Convolutional Networks: A Unified Approach to Action Segmentation
Colin Lea, Rene Vidal, Austin Reiter, Gregory D. Hager
TL;DR
Action segmentation traditionally separates local feature extraction from high-level temporal modeling, potentially losing information between the two stages. The paper introduces a hierarchical Temporal Convolutional Network that unifies these operations and reports competitive or superior results across three datasets while training much faster than an RNN-LSTM.
Problem
Existing action-segmentation systems decouple low-level spatiotemporal features from high-level temporal models, potentially losing valuable information between the two stages.
Method
The Temporal Convolutional Network hierarchically captures low-, intermediate-, and high-level temporal information using temporal convolutions, pooling or upsampling, and channel-wise normalization.
Results
The model achieves superior or competitive performance across three public action-segmentation datasets using video or sensor data, with large edit-score increases in most experiments.
Takeaways & Limitations
A unified temporal-convolutional model can support video and sensor-based action segmentation without a traditional RNN or CRF temporal model.
Abstract
from arXiv · showhide
The dominant paradigm for video-based action segmentation is composed of two steps: first, for each frame, compute low-level features using Dense Trajectories or a Convolutional Neural Network that encode spatiotemporal information locally, and second, input these features into a classifier that captures high-level temporal relationships, such as a Recurrent Neural Network (RNN). While often effective, this decoupling requires specifying two separate models, each with their own complexities, and prevents capturing more nuanced long-range spatiotemporal relationships. We propose a unified approach, as demonstrated by our Temporal Convolutional Network (TCN), that hierarchically captures relationships at low-, intermediate-, and high-level time-scales. Our model achieves superior or competitive performance using video or sensor data on three public action segmentation datasets and can be trained in a fraction of the time it takes to train an RNN.
1. Introduction
Action segmentation must simultaneously identify action intervals and classify their constituent segments, but existing systems commonly separate low-level feature extraction from high-level temporal modeling. The paper proposes unifying these stages to capture temporal information hierarchically.
- Motivation: Action segmentation assigns an action class to every frame while segmenting the video into constituent action intervals.The task supports applications including collaborative robotics and modeling activities of daily living.
- Motivation: Existing systems commonly combine short-range low-level spatiotemporal features with separate high-level temporal classifiers.Examples include IDT or ST-CNN features paired with models such as CRFs, RNNs, LSTMs, and GRUs.
- Motivation: The two-step paradigm may lose valuable information between low-level representation and high-level temporal modeling.
- Contribution: The proposed unified approach uses 1D convolutions, pooling, and channel-wise normalization to hierarchically capture low-, intermediate-, and high-level temporal information.The model updates latent activations hierarchically per layer rather than sequentially per frame.
- Contribution: The approach is broadly applicable to video and other robot sensors, and is presented as a generalization of ST-CNNs with faster training than RNN-based models.
2. Temporal Convolutional Networks (TCN)
The TCN is an encoder-decoder built from temporal convolutions, temporal pooling or upsampling, and channel-wise normalization. Its layers progressively model temporal patterns and produce per-frame action probabilities.
- Input and output: The TCN accepts sensor signals or per-frame spatial-CNN encodings as input feature vectors.The input at time t is represented by X_t with feature length F_0.
- Architecture: The encoder-decoder framework combines temporal convolutions, 1D pooling or upsampling, and channel-wise normalization.
- Encoder: Encoder filters capture how input signals evolve over an action across successive temporal layers.Each layer uses 1D filters parameterized by weights and biases.
- Encoder: Max pooling halves the temporal resolution and enables efficient computation over long periods.Channel-wise normalization is applied after each encoder pooling step.
- Decoder: The decoder replaces pooling with upsampling, repeating entries twice before convolution and normalization.
- Input and output: A learned weight matrix and bias convert the decoder representation into per-frame probabilities over C action classes.The predicted vector for frame t lies in [0, 1]^C.
- Implementation: The selected three-layer configuration uses filter counts {32, 64, 96}, a dataset-dependent filter duration, cross-entropy loss, SGD, and ADAM updates.For 50 Salads, the example filter duration is 10 seconds.
3. Evaluation
The evaluation covers three public action-segmentation datasets containing video and, in two cases, synchronized sensor data. Performance is assessed with frame accuracy and segmental edit distance.
- Datasets: The study evaluates the model on three public datasets with action-segmentation labels, video, and sensor data in two cases.
- Datasets: 50 Salads contains 50 salad-preparation sequences with video and synchronized accelerometers attached to scene objects.The evaluation uses 10 action classes at the eval granularity with five cross-validation splits.
- Datasets: JIGSAWS evaluates suturing sequences from eight users using video, robot kinematics, and 10 action classes.The protocol uses leave-one-user-out cross validation.
- Datasets: GTEA contains 28 egocentric videos of seven kitchen activities and is evaluated with 11 action classes using leave-one-user-out validation.Results are reported for user 2 to match prior evaluations.
- Metrics: Accuracy measures correctly labeled frames, while segmental edit distance measures the correctness of predicted temporal action ordering on a 0-to-100 scale.Higher edit scores are better.
4. Experiments and Discussion
Experiments compare the TCN with prior models across datasets and sensing modalities, finding strong segmentation performance and substantially faster training than an RNN-LSTM.
- The evaluation covers 50 Salads, GTEA, and JIGSAWS using video and sensor modalities, comparing spatial CNNs, a Spatiotemporal CNN, and the TCN.
- The model appears to learn temporal shifts through convolutional filters, helping address sensor-based action ambiguity and reducing over-segmentation.
- 59.7% accuracy, 47.3% edit, 0.579 mAP at 0.1, and 0.378 mAP at 0.5 were achieved on mid-level 50 Salads, exceeding Richard et al.'s reported values.
- 62.5% accuracy was achieved on GTEA under the users 1–3 setup, versus Singh et al.'s 64.4%, with high trial-to-trial variance making the difference unlikely to be statistically significant.
- Training each TCN split takes about a minute on an Nvidia Titan X, compared with about an hour for an RNN-LSTM.The reported speedup results from computing one convolutional set per layer rather than one set per time step.