Source-linked AI summary
Temporal Convolutional Networks for Action Segmentation and Detection
Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, Gregory D. Hager
TL;DR
Fine-grained action segmentation and detection require temporal models that handle subtly different actions and long-range dependencies. The paper introduces hierarchical Temporal Convolutional Networks, including encoder-decoder and dilated variants, and reports strong performance across challenging datasets, with fewer over-segmentation errors for ED-TCN. The authors conclude that TCNs are a formidable alternative to recurrent networks.
Problem
Fine-grained videos contain many subtly different actions, while prior temporal models have limited long-range modeling, attention, interpretability, or trainability.
Method
The paper introduces Temporal Convolutional Networks using hierarchical temporal convolutions, with ED-TCN combining pooling and upsampling and Dilated TCN using dilated convolutions.
Results
Across three challenging datasets, TCNs outperform strong baselines including Bi-LSTM, while ED-TCN produces fewer over-segmentation errors.
Takeaways & Limitations
TCNs capture action compositions, durations, and long-range temporal patterns, supporting their use as an alternative to recurrent networks for fine-grained action modeling.
Takeaways & Limitations
mAP can vary substantially with how confidence scores are computed, limiting its indication of real-world fine-grained action-detection performance.
Abstract
from arXiv · showhide
The ability to identify and temporally segment fine-grained human actions throughout a video is crucial for robotics, surveillance, education, and beyond. Typical approaches decouple this problem by first extracting local spatiotemporal features from video frames and then feeding them into a temporal classifier that captures high-level temporal patterns. We introduce a new class of temporal models, which we call Temporal Convolutional Networks (TCNs), that use a hierarchy of temporal convolutions to perform fine-grained action segmentation or detection. Our Encoder-Decoder TCN uses pooling and upsampling to efficiently capture long-range temporal patterns whereas our Dilated TCN uses dilated convolutions. We show that TCNs are capable of capturing action compositions, segment durations, and long-range dependencies, and are over a magnitude faster to train than competing LSTM-based Recurrent Neural Networks. We apply these models to three challenging fine-grained datasets and show large improvements over the state of the art.
1. Introduction
Fine-grained action segmentation requires modeling many subtly different actions over long videos, while prior temporal models struggle with long-range dependencies. The paper introduces Temporal Convolutional Networks that use hierarchical temporal convolutions and outperform recurrent baselines while training much faster.
- Motivation: Action segmentation identifies and classifies every action segment in videos containing many subtly different actions over minutes.The task supports applications including collaborative robotics and activity analysis.
- Limitations of prior work: Prior approaches either use sliding windows, assume segment independence, or rely on recurrent models with limited attention and difficult training.These limitations constrain modeling of long-range latent temporal patterns.
- Temporal Convolutional Networks: Temporal Convolutional Networks capture long-range patterns through a hierarchy of temporal convolutional filters.The paper presents ED-TCN and Dilated TCN variants.
- Model variants: ED-TCN combines temporal convolutions, pooling, and upsampling, whereas Dilated TCN replaces pooling and upsampling with dilated convolutions and skip connections.ED-TCN uses fewer layers with longer filters; Dilated TCN uses more layers with shorter filters.
- Results: Both TCNs capture action durations, pairwise segment transitions, and long-range temporal patterns, while outperforming the Bi-LSTM baseline and training over a magnitude faster.ED-TCN also produces fewer over-segmentation errors than other models.
- Evaluation: The paper evaluates segmentation and detection across multiple datasets and proposes a segmental F1 score intended to better reflect real-world concerns.The evaluation includes MERL Shopping and other fine-grained datasets.
2. Related Work
Action segmentation and detection address closely related temporal labeling problems, but their literature uses different temporal models and evaluation metrics. Prior work includes sliding-window detectors, segmental models, and recurrent or convolutional temporal approaches.
- Task definitions: Action segmentation predicts an action for every video frame, whereas detection outputs sparse segments with start time, end time, and class label.The two representations can be converted by adding or removing background segments.
- Action Detection: Sliding-window detection methods evaluate candidate temporal segments using spatial or spatiotemporal features and non-maximal suppression.Examples include Dense Trajectories, human pose features, and object-centric representations.
- Resources: Some dataset features and predictions were subject to release constraints, including permission requirements for MERL Shopping features.The paper states that other features would be made available.
- Action Detection: LSTM-based detection feeds per-frame CNN features into an LSTM and applies non-maximal suppression to its outputs.This approach was evaluated on the MERL Shopping dataset.
- Segmental models: Segmental models represent action transitions and durations using language-model and duration-model components.Related work also models object states at the start and end of actions on GTEA.
- Temporal convolutions: Temporal convolutional networks differ from earlier TDNNs by incorporating pooling, skip connections, newer activations, and other architectural features.The paper defines TCNs as layer-wise temporal convolutions with fixed-length receptive fields.
3. Temporal Convolutional Networks
Temporal Convolutional Networks perform framewise action prediction using temporal convolutions, with Encoder-Decoder and Dilated architectures designed to capture long-range patterns. The models use pooling or dilation, residual and skip connections, and fixed receptive fields to model temporal structure.
- Temporal Convolutional Networks: TCNs compute all time steps layer-wise with temporal convolutions, and each prediction depends on a fixed-length receptive field.The input consists of per-frame video features, while outputs are action-label probabilities for each frame.
- Encoder-Decoder TCN: The Encoder-Decoder TCN combines temporal convolutions and max pooling in its encoder with upsampling and convolutions in its decoder.Pooling condenses temporal representations, while decoder convolutions distribute them back to frame-level action predictions.
- Encoder-Decoder TCN: Decoder convolutions provide a large performance improvement and appear to capture pairwise transitions between actions.The decoder reverses the encoder’s layer indexing so its final representations produce frame-level predictions.
- Dilated TCN: The Dilated TCN uses blocks of dilated convolutions, residual connections, and skip connections to expand its receptive field without drastically increasing parameters.Its dilation rate increases across consecutive layers, while the block outputs are summed through skip connections.
- Dilated TCN: The Dilated TCN’s receptive field is r(B, L) = B ∗2^L, where B is the number of blocks and L is the number of layers per block.Its smaller filters require more layers or blocks than the Encoder-Decoder TCN for an equal-sized receptive field.
- Prediction settings: The models support causal and acausal prediction settings, determining whether predictions use only past data or can use the full sequence.Causal prediction at time t uses data from times 1 to t, while acausal prediction may use future time steps.
4. Evaluation & Discussion
The evaluation combines synthetic tests, three challenging datasets, cross-community metrics, and hyperparameter analyses to assess TCN temporal modeling. ED-TCN generally delivers the strongest segmentation quality, while TCNs train substantially faster than Bi-LSTM.
- Evaluation metrics: Frame-wise accuracy can conceal substantial over-segmentation differences that matter for human-robot interaction and video summarization.Models with similar accuracy may have very different F1@k and edit scores.
- Evaluation metrics: A segmental F1 score evaluates both segmentation and detection while penalizing over-segmentation, tolerating minor temporal shifts, and avoiding dependence on action duration.It determines true and false positives using segment IoU and computes precision, recall, and F1 across classes.
- Synthetic experiments: TCNs perfectly segment synthetic actions whose subactions share identical features, indicating sensitivity to action composition and class-specific duration.The result relies on their long receptive fields and fixed but differing action durations.
- Synthetic experiments: TCNs remain effective under delayed inputs: at s = 5 they detect all but the first and last actions, while Bi-LSTM performance degrades faster as delay increases.With no delay, all models predict perfectly; TCN performance remains strong to roughly half the receptive-field length.
- Dataset results: ED-TCN outperforms all other models across both action granularities and all reported metrics on 50 Salads.For mid-level detection, ED-TCN reaches 64.9 mAP@10 and 42.3 mAP@50, compared with Richard et al.'s 37.9 mAP@10 and 22.9 mAP@50.
- Dataset results: On MERL Shopping, midpoint mAP can favor short sparse detections even when predicted start and stop times are poor, whereas acausal TCNs outperform causal variants.Dilated TCN has better midpoint mAP, while ED-TCN has better F1 scores in both causal and acausal settings.
- Dataset results: ED-TCN performs on par with Singh et al.'s ensemble on Georgia Tech Egocentric Activities while using simpler spatial CNN features computable in real time.The compared ensemble combines EgoNet features with TDD and does not incorporate a temporal model.
- Ablations and efficiency: ED-TCN's longer filters are associated with stronger F1 performance and fewer over-segmentation errors, whereas Dilated TCN performs well on accuracy but is less robust to over-segmentation.The paper attributes this difference likely to the Dilated TCN's shorter filter lengths.
5. Conclusion
Temporal Convolutional Networks use hierarchical convolutions to capture long-range temporal patterns in fine-grained action sequences. Across synthetic and challenging real-world datasets, they captured complex temporal structure, handled delays, outperformed strong baselines, and achieved state-of-the-art performance.
- TCNs use a hierarchy of convolutions to capture long-range temporal patterns.
- TCNs captured complex patterns including action compositions and durations on synthetic data.
- TCNs were robust to time-delays in synthetic experiments.
- TCNs outperformed strong baselines, including Bidirectional LSTM, on challenging datasets.
- TCNs achieved state-of-the-art performance on challenging datasets.