PaperlayerView exact revision on arXiv

Source-linked AI summary

MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation

Yazan Abu Farha, Juergen Gall

arXiv:1903.01945v2cs.CV

TL;DR

Long untrimmed-video action segmentation requires locating and classifying actions over time. The paper proposes MS-TCN, a full-resolution multi-stage temporal-convolutional model with smoothing loss, and reports state-of-the-art results on three datasets.

  • Problem

    Action segmentation in long untrimmed videos remains challenging, while weakly supervised approaches are outside this paper’s fully supervised scope.

  • Method

    MS-TCN uses multiple stages of dilated temporal convolutions whose predictions are successively refined and trained with a smoothing loss.

  • Results

    The model outperforms state-of-the-art methods across three datasets and three metrics, with up to 12.6% higher frame-wise accuracy on 50Salads.

  • Takeaways & Limitations

    The fully convolutional architecture captures temporal dependencies, reduces over-segmentation errors, and is efficient during training and testing.

Abstract

from arXiv · show

Temporally locating and classifying action segments in long untrimmed videos is of particular interest to many applications like surveillance and robotics. While traditional approaches follow a two-step pipeline, by generating frame-wise probabilities and then feeding them to high-level temporal models, recent approaches use temporal convolutions to directly classify the video frames. In this paper, we introduce a multi-stage architecture for the temporal action segmentation task. Each stage features a set of dilated temporal convolutions to generate an initial prediction that is refined by the next one. This architecture is trained using a combination of a classification loss and a proposed smoothing loss that penalizes over-segmentation errors. Extensive evaluation shows the effectiveness of the proposed model in capturing long-range dependencies and recognizing action segments. Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.

1. Introduction

Long untrimmed-video action segmentation remains challenging despite successful short-video classification. MS-TCN addresses this with full-resolution multi-stage temporal convolutions, refined predictions, and smoothing-based training.

  • Action segmentation in long untrimmed videos remains difficult, although short trimmed-video classification has achieved strong results.
  • Earlier methods used sliding windows or frame-wise classifiers combined with high-level temporal models, but required costly optimization over long sequences.
  • Recent temporal-convolutional approaches capture long-range dependencies but operate at only a few frames per second because of temporal resolution reduction.
  • MS-TCN uses multiple stages in which each prediction is refined by the next, with dilated 1D convolutions applied at each stage.
  • A smoothing loss penalizes over-segmentation errors, and evaluation reports state-of-the-art results on 50Salads, GTEA, and Breakfast.

2. Related Work

Prior action-segmentation research used windowed detection, probabilistic temporal models, recurrent architectures, and temporal convolutions. MS-TCN differs by targeting fully supervised segmentation at full temporal resolution.

  • Traditional approaches include sliding windows, hidden Markov models, Bayes networks, grammars, and dynamic programming for temporal action recognition.
  • The related work spans both traditional probabilistic pipelines and newer temporal-convolutional approaches for action segmentation.
  • Other systems combine frame or chunk representations with recurrent models such as bidirectional LSTMs to capture dependencies across time.
  • Temporal-convolutional methods adapt encoder-decoder architectures from speech synthesis, using pooling and upsampling to model video sequences.
  • Weakly supervised methods learn from transcripts or alignments, whereas this paper addresses temporal action segmentation in a fully supervised setup.

3. Temporal Action Segmentation

MS-TCN performs frame-wise action segmentation with sequential temporal-convolution stages that refine predictions and use smoothing to reduce over-segmentation. Its dilated convolutions provide broad temporal context while preserving full temporal resolution.

  • 3. Temporal Action Segmentation: MS-TCN infers a class label for every frame in a video using a multi-stage temporal convolutional network.The first stage receives frame-wise video features, while later stages operate on predictions from preceding stages.
  • 3.1. Single-Stage TCN: Each stage uses dilated 1D convolutions and residual connections to obtain a large temporal receptive field with relatively few parameters.The single-stage model uses acausal convolutions with kernel size 3 and exponentially increasing dilation factors.
  • 3.2. Multi-Stage TCN: Sequential stages refine preceding frame-wise predictions, providing additional context and helping model dependencies between action classes and plausible action sequences.Later stages receive only frame-wise probabilities, without additional features.
  • 3.3. Loss Function: The model combines cross-entropy classification with a truncated mean-squared-error smoothing loss over frame-wise log-probabilities.The smoothing term is intended to reduce over-segmentation errors, and losses are summed across stages during training.
  • 3. Temporal Action Segmentation: The reported configuration uses four stages with ten dilated convolution layers per stage, 64 filters, filter size 3, and dropout after each layer.Training uses Adam with learning rate 0.0005, τ = 4, and λ = 0.15.

4. Experiments

The model is evaluated on three challenging datasets using frame-wise features and metrics that emphasize segment-level prediction quality. The experiments include 50Salads stage comparisons and report accuracy, edit distance, and segmental F1.

  • Datasets: The model is evaluated on 50Salads, GTEA, and Breakfast, covering salad preparation, egocentric activities, and breakfast-related activities.The datasets contain 50, 28, and 1,712 videos, respectively.
  • Experimental Setup: For GTEA and Breakfast, features are used at 15 fps, while 50Salads features are downsampled from 30 fps to 15 fps.I3D features are extracted for the video frames.
  • Evaluation Metrics: Evaluation uses frame-wise accuracy, segmental edit distance, and F1 scores at 10%, 25%, and 50% overlap thresholds.The overlap threshold is based on intersection over union.
  • Evaluation Metrics: Segmental F1 is included because frame-wise accuracy gives greater weight to long actions and is less sensitive to over-segmentation errors.The paper adopts segmental F1 as a measure of prediction quality.
  • Stage Comparison: Table 1 compares a single-stage model with multi-stage models using different numbers of stages on 50Salads.The accompanying qualitative result is shown in Figure 3.

4.1. Effect of the Number of Stages

The experiments show that multi-stage refinement improves segment-level predictions beyond a single-stage model, and that the gain is not explained solely by adding parameters. A four-stage multi-stage model outperforms a parameter-matched deep single-stage TCN by up to 27%.

  • Effect of the Number of Stages: Multi-stage models have comparable frame-wise accuracy but substantially better segmental edit and F1 scores than the single-stage model.The single-stage model produces many over-segmentation errors, reflected by its low F1 score.
  • Multi-Stage TCN vs. Deeper Single-Stage TCN: A parameter-matched single-stage TCN uses 48 layers, matching the layer count of a four-stage MS-TCN.Each MS-TCN stage has 12 layers.
  • Loss Comparison: The qualitative comparison in Figure 4 examines predictions produced with different loss functions on 50Salads.Table 3 provides the corresponding quantitative loss comparison.

4.3. Comparing Different Loss Functions

The proposed loss combines classification with smoothing over frame-wise log-probabilities to reduce over-segmentation. It improves segmental metrics over cross-entropy alone and outperforms KL divergence in the reported comparison.

  • Loss Design: The proposed loss combines cross-entropy with truncated mean squared error over frame-wise log-probabilities to produce smoother predictions.The smoothing term is added to reduce over-segmentation errors.
  • Results: 5%: the proposed loss achieves an absolute improvement in F1 and edit scores over the comparison loss in Table 3.The smoothing loss only slightly improves frame-wise accuracy but produces fewer over-segmentation errors.
  • Loss Surface: Figure 5 compares the KL divergence and truncated mean squared loss surfaces for two classes.The plotted probabilities are the predicted probability and the preceding target probability for each class.
  • Loss Design: The proposed loss performs better than KL divergence because it penalizes small prediction-target differences and is symmetric.A symmetric KL variant also performs worse than the proposed loss.

4.4. Impact of λ and τ

The smoothing loss is controlled by λ and τ, whose effects are examined experimentally. The reported analysis indicates that λ has little impact on performance, while the supplied passages do not state a complete result for τ.

  • Hyper-parameters: The smoothing loss has two hyper-parameters, λ and τ, whose effects are evaluated experimentally.The paper sets λ = 0.15 in all experiments discussed in the analysis.
  • Impact of λ: λ = 0.15: this is the value used in all experiments reported in the supplied analysis.Reducing λ to 0.05 still improves performance, but not as well as the default setting.
  • Impact of λ: The impact of λ on performance is very small across the tested values.The passage reports that λ = 0.05 still improves performance, though less than the default.
  • Additional Analyses: Table 4 reports the impact of λ and τ on the 50Salads dataset, while Table 5 examines passing features to higher stages.Figure 6 provides qualitative results for the effect of passing features to higher stages.

4.5. Effect of Passing Features to Higher Stages

The experiments examine how temporal resolution and higher-stage inputs affect multi-stage TCN performance. The model handles both low and high resolutions, while higher resolution favors frame accuracy and lower resolution favors edit distance and segmental F1.

  • Feature Inputs: The experiment compares multi-stage TCNs using only frame-wise probabilities versus probabilities combined with additional features at higher stages.The supplied passage introduces this comparison, but does not report its outcome.
  • Temporal Resolution: 15 fps is the higher temporal resolution the approach can handle, compared with previous models operating at 1–3 fps.The experiment also evaluates the model at 1 fps.
  • Temporal Resolution: Lower temporal resolution improves edit distance and segmental F1, whereas higher resolution improves frame-wise accuracy.The passage attributes reduced over-segmentation susceptibility to operating at low temporal resolution.

4.7. Impact of the Number of Layers

The experiments assess model depth, video duration, and feature fine-tuning. More layers expand the receptive field and improve performance up to the tested range, while longer videos remain somewhat harder and fine-tuning provides smaller gains for segmentation.

  • Impact of the Number of Layers: Increasing the number of layers per stage from 6 to 10 significantly improves performance through a larger receptive field.Using 12 layers does not improve frame-wise accuracy but slightly increases F1 scores.
  • Video Duration: The model performs well on both short and long GTEA videos, but performance is slightly worse on longer videos because of the limited receptive field.The evaluation groups GTEA videos by duration.
  • Feature Fine-Tuning: Fine-tuning improves results, but its effect on action segmentation is lower than on action recognition.The multi-stage architecture outperforms the single-stage architecture with and without fine-tuning.

4.9. Comparison with the State-of-the-Art

Against state-of-the-art methods, MS-TCN performs better across three datasets and three evaluation metrics, while fully convolutional computation also makes it faster than the compared recurrent baseline. Appearance features contribute little on Breakfast.

  • Benchmark Comparison: MS-TCN outperforms state-of-the-art methods on 50Salads, GTEA, and Breakfast across F1 score, segmental edit distance, and frame-wise accuracy.The reported frame-wise accuracy margin reaches up to 12.6% on 50Salads.
  • Benchmark Comparison: 12.6% is the largest reported frame-wise accuracy margin on the 50Salads dataset.The comparison covers three datasets and three evaluation metrics.
  • Feature Comparison: On Breakfast, I3D and IDT features have a small overall impact, with I3D slightly better for accuracy and edit distance but IDT better for F1 score.The passage explains that I3D encodes motion and appearance, while the supplied text truncates the corresponding IDT explanation.
  • Efficiency: Training a four-stage MS-TCN for 50 epochs is four times faster than training a single 64-dimensional Bi-LSTM cell on one GTX 1080 Ti.MS-TCN computes activations for all time steps in parallel, unlike the sequential LSTM recurrence.

5. Conclusion

The paper concludes that its multi-stage, fully convolutional architecture captures temporal dependencies, reduces over-segmentation, and achieves strong results efficiently. Dilated convolutions enlarge the receptive field, while smoothing loss further improves prediction quality.

  • Conclusion: The multi-stage architecture captures temporal dependencies between action classes and reduces over-segmentation errors.The conclusion attributes these capabilities to the experimental evaluation.
  • Conclusion: Dilated convolutions increase the temporal receptive field instead of using the commonly used temporal pooling.The architecture is fully convolutional rather than recurrent.
  • Conclusion: The smoothing loss provides an additional improvement in prediction quality.It is introduced alongside the multi-stage architecture.
  • Conclusion: The model outperforms state-of-the-art methods on three challenging datasets with a large margin and is efficient during training and testing.The conclusion attributes efficiency to the model being fully convolutional.
Loading 1903.01945v2…