PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation

Nadine Behrmann, S. Alireza Golestaneh, Zico Kolter, Juergen Gall, Mehdi Noroozi

arXiv:2209.00638v2cs.CV

TL;DR

Frame-level action prediction can over-segment videos, especially with small datasets, while action segmentation must map long frame sequences to concise segment sequences. UVAST uses an autoregressive Transformer with auxiliary frame-wise supervision and an alignment decoder, and extends the framework to timestamp supervision with constrained k-medoids pseudo-segmentations. It achieves competitive frame-wise accuracy, substantially outperforms other approaches in Edit distance, and reaches state-of-the-art segmental F1 with post-processing on three benchmarks.

  • Problem

    Frame-level action prediction can over-segment videos, particularly when training data is relatively small and frame-level annotations are expensive.

    Source: §1 · p. 1

  • Method

    UVAST uses an autoregressive Transformer to predict action segments, adds frame-wise encoder supervision and an alignment decoder, and uses constrained k-medoids to produce pseudo-segmentations from timestamps.

    Source: §1 · pp. 2–3, §3.1 · p. 5

  • Results

    Across 50Salads, GTEA, and Breakfast, UVAST has competitive frame-wise accuracy, substantially outperforms other approaches in Edit distance, and achieves state-of-the-art segmental F1 with Viterbi or FIFA post-processing.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The segment-level Transformer framework achieves competitive performance on all three datasets and applies to both fully and timestamp-supervised action segmentation.

    Source: §1 · p. 3, §5 · p. 16

    On Breakfast, frame-wise accuracy is lower than state of the art, and UVAST localizes segment boundaries less accurately than its Edit and F1 scores suggest.

    Source: Fully Supervised Comparison · p. 12

Abstract

from arXiv · show

This paper introduces a unified framework for video action segmentation via sequence to sequence (seq2seq) translation in a fully and timestamp supervised setup. In contrast to current state-of-the-art frame-level prediction methods, we view action segmentation as a seq2seq translation task, i.e., mapping a sequence of video frames to a sequence of action segments. Our proposed method involves a series of modifications and auxiliary loss functions on the standard Transformer seq2seq translation model to cope with long input sequences opposed to short output sequences and relatively few videos. We incorporate an auxiliary supervision signal for the encoder via a frame-wise loss and propose a separate alignment decoder for an implicit duration prediction. Finally, we extend our framework to the timestamp supervised setting via our proposed constrained k-medoids algorithm to generate pseudo-segmentations. Our proposed framework performs consistently on both fully and timestamp supervised settings, outperforming or competing state-of-the-art on several datasets. Our code is publicly available at https://github.com/boschresearch/UVAST.

1 Introduction

UVAST maps video frames to action segments with an autoregressive Transformer, addressing over-segmentation in frame-level prediction. Auxiliary frame-wise supervision, an alignment decoder, and timestamp-derived pseudo-segmentations support the approach across supervision settings.

1 Introduction

  • UVAST maps video frames directly to action-label and duration pairs, with an autoregressive decoder predicting the higher-level segment sequence.Unlike frame-level prediction, this segment-level approach is less prone to over-segmentation.Source: §1 · p. 2
  • Long sequences of similar frames, short segment outputs, and relatively few training videos make it difficult for a standard seq2seq model to retain the information needed for segmentation.The paper identifies these differences from language translation as motivation for additional model modifications and loss terms.Source: §1 · p. 2
  • A frame-wise encoder loss improves transcript prediction, while an alignment decoder fuses decoder and encoder features to address weak duration localization.The encoder’s frame-wise predictions localize actions well but tend to over-segment; the decoder predicts transcripts more effectively than durations.Source: §1 · pp. 2–3
  • Up to 81% frame-wise accuracy on the training set is achieved by constrained k-medoids pseudo-segmentations generated from timestamp supervision.The generated frame labels can train UVAST and can also be paired with frame-wise prediction methods such as ASFormer.Source: §1 · p. 3
  • On 50Salads, GTEA, and Breakfast, UVAST has competitive frame-wise accuracy, substantially better Edit distance, and state-of-the-art segmental F1 with Viterbi or FIFA.The Edit distance comparison measures action-sequence prediction.Source: §1 · p. 3

2 Related Work

Prior work spans frame-level fully supervised methods and weaker transcript- or timestamp-supervised approaches. Transformer-based action segmentation includes frame-level predictors such as ASFormer, whereas UVAST predicts segments autoregressively.

2 Related Work

  • Fully supervised methods progress from sliding windows and HMMs to temporal convolutional networks, while frame-level predictors remain prone to over-segmentation and often need refinement or smoothing.Temporal pooling can also lose fine-grained temporal information; later multi-stage TCNs preserve higher temporal resolution.Source: §2 · p. 3
  • Transcript supervision reduces annotation cost but can suffer in performance; timestamp supervision has comparable annotation cost and supplies rough segment locations.Prior transcript-based methods include pseudo-labeling and transcript-to-frame alignment approaches.Source: §2 · p. 4
  • Standard Transformers face difficulty with very long sequences because self-attention scales quadratically, and ASFormer predicts frame labels rather than UVAST’s autoregressive action segments.UVAST uses an encoder similar to ASFormer but a different decoder design.Source: §2 · p. 4

3 Method

UVAST reframes action segmentation as direct prediction of action segments from video frames, addressing the over-segmentation associated with frame-level predictions. Its Transformer framework adds encoder supervision, duration alignment, and timestamp-derived pseudo-segmentations.

3 Method

  • UVAST directly predicts action segments, an alternative representation intended to overcome the over-segmentation problem of frame-level prediction.The output represents segments as action-label and duration pairs.Source: §3 · pp. 4–5, §3.1 · p. 5

3.1 Transformer for Auto-Regressive Segment Prediction

  • A Transformer encoder processes frame features, while an autoregressive decoder predicts the ordered action transcript from those features.The decoder uses cross-attention to the frame features; a separate module predicts durations because direct decoder duration prediction has low localization performance.Source: §3.1 · pp. 5–6

3.2 Training Objective

  • Training combines frame-wise encoder and segment-wise decoder cross-entropy with group-wise losses over predictions averaged by ground-truth action class.The frame-wise loss gives direct feedback to the encoder, while the segment-wise loss trains decoder predictions.Source: §3.2 · p. 6

3.3 Cross-Attention Loss

  • Cross-attention loss provides location feedback to the transcript decoder, while a separate alignment decoder assigns frames to segments for implicit duration prediction.The alignment decoder is trained in a separate stage on frozen encoder and decoder features; this produces a more robust model that suffers less from overfitting.Source: §3.3 · pp. 7–8

3.4 Timestamp Supervision

  • For timestamp supervision, constrained k-medoids propagates timestamps into frame-level annotations by enforcing temporally continuous clusters.The method initializes clusters with timestamp supervision; the paper reports that input features alone give surprisingly accurate segmentations.Source: §3.4 · pp. 8–9

4 Experiments

UVAST is evaluated on three action-segmentation datasets under fully and timestamp supervision, with strong Edit and F1 performance reported across both settings. The experiments also examine its training choices, qualitative segmentations, and loss components.

4.1 Datasets

  • Across Breakfast, GTEA, and 50Salads, evaluation uses four-fold cross-validation for Breakfast and GTEA and five-fold for 50Salads.The datasets are cited as 50Salads, GTEA, and Breakfast.Source: §4.1 · p. 9

4.2 Evaluation Metrics

  • Frame-wise accuracy favors long actions and is less sensitive to over-segmentation, while Edit and F1 provide more comprehensive measures of segmentation quality.The evaluation reports Acc, Edit, and F1 at IoU thresholds of 10%, 25%, and 50%.Source: §4.2 · pp. 9–10

4.3 Implementation Details and Training

  • Training uses batch size 1, Adam at learning rate 0.0005, and at most 800 epochs; longer segments are split during training but not inference.The separate alignment decoder is trained with the encoder-decoder frozen, using a single-layer, single-head decoder.Source: §4.3 · p. 10

4.4 Performance Evaluation

  • With Viterbi or FIFA, UVAST outperforms state of the art in Edit and F1 on all three fully supervised datasets.Transcript prediction without duration also achieves better Edit scores than existing methods on all three datasets.Source: Table 1 · p. 11, Fully Supervised Comparison · p. 11
  • Timestamp-supervised UVAST achieves the best results on all datasets and metrics except Breakfast accuracy, with Edit performance comparable to full supervision.The constrained k-medoids pseudo-labels also let ASFormer and MS-TCN perform reasonably well in this setting.Source: Table 2 · p. 12, Timestamp Supervision Comparison · p. 13
  • On Breakfast, ASFormer generally has higher frame-wise accuracy while UVAST has better Edit; lower accuracy alongside higher Edit/F1 indicates less accurate boundary localization.The paper identifies improving segment boundaries, potentially with refinement like ASFormer, as a research direction.Source: Fully Supervised Comparison · p. 12
  • Constrained k-medoids uses input features and ground-truth timestamps to form continuous pseudo-segments, which supervise the encoder during timestamp-supervised training.The authors report that input features already give surprisingly good pseudo-segmentation performance.Source: Timestamp Supervision Comparison · p. 12

4.5 Qualitative Evaluation

  • In qualitative examples, encoder frame predictions show over-segmentation, while decoder predictions produce coherent action segments.Figure 4 compares ground truth and predicted segmentations for fully and timestamp-supervised Breakfast videos.Source: Figure 4 · p. 13, §4.5 · p. 13

5 Conclusion

UVAST unifies fully and timestamp supervised temporal action segmentation in a Transformer seq2seq design. Its segment-level predictions address over-segmentation, while three proposed approaches support competitive performance on all three datasets.

  • Explicit segment-duration prediction does not work out of the box, so the authors propose three approaches to address this challenge.Source: §5 · p. 16
  • UVAST’s segment-level predictions effectively address the over-segmentation problem.Source: §5 · p. 16
  • The proposed approaches enable competitive performance on all three datasets.Source: §5 · p. 16

Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation Supplementary Material

The supplementary material covers dataset characteristics, architecture and algorithm studies, hyper-parameters, and grouping losses.

1 Datasets

The experiments use GTEA, 50Salads, and Breakfast, which differ in dataset scale, video length, and action-segment structure.

2 Implementation Details

The implementation details compare encoder and segment-duration training choices, describe inference and clustering considerations, and report supplementary ablations.

Impact of Encoder Model

Impact of Split-Segment

Hyper-Parameters

  • For 50Salads, cross-attention smoothing averages the attention map along T to reduce noise associated with long videos and few training samples.Source: Impact of Encoder Model · p. 21

K-Medoids

3 Grouping Loss Terms

Because action-class frequencies and frame durations are imbalanced, the section compares modified cross-entropy losses that average predictions by class. The reported results favor different variants for segment-wise decoder loss and frame-wise classification.

3 Grouping Loss Terms

  • The two cross-entropy modifications average each class’s probabilities or its logits before applying Softmax, addressing imbalanced action classes.The datasets’ action frequencies and the number of frames occupied by each action influence encoder and decoder cross-entropy.Source: §3 · pp. 22–23
  • Frames are grouped by their class labels, and the class-wise predicted probabilities or logits are averaged within each group.The passage defines frame logits and their corresponding Softmax probabilities before describing the class-wise averages.Source: §3 · p. 23
  • The three compared losses comprise element-wise cross-entropy and two group-wise cross-entropies, averaging outside or inside Softmax, respectively.The table reports Edit results for the first stage of training.Source: §3 · p. 23, Table 13 · p. 24
  • The probability-averaging loss works best for segment-wise decoder loss, while the logit-averaging loss works best for frame-wise classification.These are first-stage transcript-prediction results on split 1 of 50Salads.Source: §3 · p. 23, Table 13 · p. 24
Loading 2209.00638v2…