PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Convolutional Networks: A Unified Approach to Action Segmentation

Colin Lea, Rene Vidal, Austin Reiter, Gregory D. Hager

arXiv:1608.08242v1cs.CV

TL;DR

Action segmentation traditionally separates local feature extraction from high-level temporal modeling, potentially losing information between the two stages. The paper introduces a hierarchical Temporal Convolutional Network that unifies these operations and reports competitive or superior results across three datasets while training much faster than an RNN-LSTM.

  • Problem

    Existing action-segmentation systems decouple low-level spatiotemporal features from high-level temporal models, potentially losing valuable information between the two stages.

    Source: §1 · p. 1

  • Method

    The Temporal Convolutional Network hierarchically captures low-, intermediate-, and high-level temporal information using temporal convolutions, pooling or upsampling, and channel-wise normalization.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    The model achieves superior or competitive performance across three public action-segmentation datasets using video or sensor data, with large edit-score increases in most experiments.

    Source: §4 · p. 3

  • Takeaways & Limitations

    A unified temporal-convolutional model can support video and sensor-based action segmentation without a traditional RNN or CRF temporal model.

    Source: §1 · pp. 1–2, §4 · p. 3

Abstract

from arXiv · show

The dominant paradigm for video-based action segmentation is composed of two steps: first, for each frame, compute low-level features using Dense Trajectories or a Convolutional Neural Network that encode spatiotemporal information locally, and second, input these features into a classifier that captures high-level temporal relationships, such as a Recurrent Neural Network (RNN). While often effective, this decoupling requires specifying two separate models, each with their own complexities, and prevents capturing more nuanced long-range spatiotemporal relationships. We propose a unified approach, as demonstrated by our Temporal Convolutional Network (TCN), that hierarchically captures relationships at low-, intermediate-, and high-level time-scales. Our model achieves superior or competitive performance using video or sensor data on three public action segmentation datasets and can be trained in a fraction of the time it takes to train an RNN.

1. Introduction

Action segmentation must simultaneously identify action intervals and classify their constituent segments, but existing systems commonly separate low-level feature extraction from high-level temporal modeling. The paper proposes unifying these stages to capture temporal information hierarchically.

Motivation

  • Action segmentation assigns an action class to every frame while segmenting the video into constituent action intervals.The task supports applications including collaborative robotics and modeling activities of daily living.Source: §1 · p. 1
  • Existing systems commonly combine short-range low-level spatiotemporal features with separate high-level temporal classifiers.Examples include IDT or ST-CNN features paired with models such as CRFs, RNNs, LSTMs, and GRUs.Source: §1 · pp. 1–2
  • The two-step paradigm may lose valuable information between low-level representation and high-level temporal modeling.Source: §1 · p. 1

Contribution

  • The proposed unified approach uses 1D convolutions, pooling, and channel-wise normalization to hierarchically capture low-, intermediate-, and high-level temporal information.The model updates latent activations hierarchically per layer rather than sequentially per frame.Source: §1 · p. 1
  • The approach is broadly applicable to video and other robot sensors, and is presented as a generalization of ST-CNNs with faster training than RNN-based models.Source: §1 · p. 2

2. Temporal Convolutional Networks (TCN)

The TCN is an encoder-decoder built from temporal convolutions, temporal pooling or upsampling, and channel-wise normalization. Its layers progressively model temporal patterns and produce per-frame action probabilities.

Input and output

  • The TCN accepts sensor signals or per-frame spatial-CNN encodings as input feature vectors.The input at time t is represented by X_t with feature length F_0.Source: §2 · p. 2

Architecture

  • The encoder-decoder framework combines temporal convolutions, 1D pooling or upsampling, and channel-wise normalization.Source: §2 · p. 2

Encoder

  • Encoder filters capture how input signals evolve over an action across successive temporal layers.Each layer uses 1D filters parameterized by weights and biases.Source: §2 · p. 2
  • Max pooling halves the temporal resolution and enables efficient computation over long periods.Channel-wise normalization is applied after each encoder pooling step.Source: §2 · p. 2

Decoder

  • The decoder replaces pooling with upsampling, repeating entries twice before convolution and normalization.Source: §2 · p. 2

Input and output

  • A learned weight matrix and bias convert the decoder representation into per-frame probabilities over C action classes.The predicted vector for frame t lies in [0, 1]^C.Source: §2 · p. 2

Implementation

  • The selected three-layer configuration uses filter counts {32, 64, 96}, a dataset-dependent filter duration, cross-entropy loss, SGD, and ADAM updates.For 50 Salads, the example filter duration is 10 seconds.Source: §2 · p. 3

3. Evaluation

The evaluation covers three public action-segmentation datasets containing video and, in two cases, synchronized sensor data. Performance is assessed with frame accuracy and segmental edit distance.

Datasets

  • The study evaluates the model on three public datasets with action-segmentation labels, video, and sensor data in two cases.Source: §3 · p. 3
  • 50 Salads contains 50 salad-preparation sequences with video and synchronized accelerometers attached to scene objects.The evaluation uses 10 action classes at the eval granularity with five cross-validation splits.Source: §3 · p. 3
  • JIGSAWS evaluates suturing sequences from eight users using video, robot kinematics, and 10 action classes.The protocol uses leave-one-user-out cross validation.Source: §3 · p. 3
  • GTEA contains 28 egocentric videos of seven kitchen activities and is evaluated with 11 action classes using leave-one-user-out validation.Results are reported for user 2 to match prior evaluations.Source: §3 · p. 3

Metrics

  • Accuracy measures correctly labeled frames, while segmental edit distance measures the correctness of predicted temporal action ordering on a 0-to-100 scale.Higher edit scores are better.Source: §3 · p. 3

4. Experiments and Discussion

Experiments compare the TCN with prior models across datasets and sensing modalities, finding strong segmentation performance and substantially faster training than an RNN-LSTM.

  • The evaluation covers 50 Salads, GTEA, and JIGSAWS using video and sensor modalities, comparing spatial CNNs, a Spatiotemporal CNN, and the TCN.Source: §4 · p. 3
  • The model appears to learn temporal shifts through convolutional filters, helping address sensor-based action ambiguity and reducing over-segmentation.Source: §4 · p. 3
  • 59.7% accuracy, 47.3% edit, 0.579 mAP at 0.1, and 0.378 mAP at 0.5 were achieved on mid-level 50 Salads, exceeding Richard et al.'s reported values.Source: §4 · p. 3
  • 62.5% accuracy was achieved on GTEA under the users 1–3 setup, versus Singh et al.'s 64.4%, with high trial-to-trial variance making the difference unlikely to be statistically significant.Source: §4 · p. 3
  • Training each TCN split takes about a minute on an Nvidia Titan X, compared with about an hour for an RNN-LSTM.The reported speedup results from computing one convolutional set per layer rather than one set per time step.Source: §4 · p. 3

Related papers

Loading 1608.08242v1…