PaperlayerView exact revision on arXiv

Source-linked AI summary

ASFormer: Transformer for Action Segmentation

Fangqiu Yi, Hongyu Wen, Tingting Jiang

arXiv:2110.08568v1cs.CV

TL;DR

Action segmentation requires framewise labeling of long videos, while vanilla Transformers lack suitable inductive bias, long-sequence handling, and refinement-oriented decoders. ASFormer combines local temporal convolutions, hierarchical attention, and iterative cross-attention decoders; experiments on three public datasets report state-of-the-art results.

  • Problem

    Vanilla Transformers face insufficient inductive bias for small training sets, difficulty representing thousands-frame videos, and decoder limitations for refining action predictions.

  • Method

    ASFormer uses local temporal convolutions, a predefined hierarchical representation pattern, and successive cross-attention decoders to refine encoder predictions.

  • Results

    State-of-the-art action-segmentation results are reported on the 50Salads, Breakfast, and GTEA datasets.

  • Takeaways & Limitations

    The results demonstrate ASFormer's effectiveness as a Transformer-based solution for action segmentation on three public datasets.

Abstract

from arXiv · show

Algorithms for the action segmentation task typically use temporal models to predict what action is occurring at each frame for a minute-long daily activity. Recent studies have shown the potential of Transformer in modeling the relations among elements in sequential data. However, there are several major concerns when directly applying the Transformer to the action segmentation task, such as the lack of inductive biases with small training sets, the deficit in processing long input sequence, and the limitation of the decoder architecture to utilize temporal relations among multiple action segments to refine the initial predictions. To address these concerns, we design an efficient Transformer-based model for action segmentation task, named ASFormer, with three distinctive characteristics: (i) We explicitly bring in the local connectivity inductive priors because of the high locality of features. It constrains the hypothesis space within a reliable scope, and is beneficial for the action segmentation task to learn a proper target function with small training sets. (ii) We apply a pre-defined hierarchical representation pattern that efficiently handles long input sequences. (iii) We carefully design the decoder to refine the initial predictions from the encoder. Extensive experiments on three public datasets demonstrate that effectiveness of our methods. Code is available at \url{https://github.com/ChinaYi/ASFormer}.

1 Introduction

Action segmentation assigns an action label to every frame of a minutes-long untrimmed video, but vanilla Transformers face three task-specific concerns. ASFormer addresses them with local connectivity, hierarchical representations, and iterative decoder refinement.

  • Motivation: Action segmentation labels every frame in minutes-long untrimmed videos using pre-extracted frame-wise features.Unlike action classification, it models temporal relations across the sequence.
  • Challenges: Vanilla Transformers lack inductive bias for small action-segmentation training sets, making target-function learning difficult over a broad hypothesis space.The paper links this concern to the relatively small size of action-segmentation datasets.
  • Challenges: Self-attention struggles with videos containing thousands of frames because attention weights are initially nearly uniform and layers may not cooperate effectively.The concern is forming an effective representation over long input sequences.
  • Challenges: Vanilla Transformer decoders are not designed to refine initial predictions using temporal relations among multiple action segments.Prior work instead applied additional TCNs or GCNs for refinement.
  • ASFormer: ASFormer introduces local temporal convolutions, a hierarchical self-attention pattern, and decoders that iteratively refine encoder predictions.The encoder uses dilated temporal convolution, while the decoder uses cross-attention to bring in encoder information.
  • Results: Experiments on 50Salads, Breakfast, and GTEA report state-of-the-art action-segmentation results.The reported results emphasize accurate and smooth predictions.

2 Related Work

Prior action-segmentation research progressed from proposal-based and probabilistic sequence models to temporal convolutions, graph reasoning, domain adaptation, boundary detection, and coarse-to-fine temporal modeling.

  • Action Segmentation: Earlier approaches used sliding windows with non-maximum suppression, Conditional Random Fields, Markov models, and recurrent neural networks.These methods classified framewise actions or detected action segments.
  • Recent Methods: Later work introduced graph-based temporal reasoning, unlabeled-video domain adaptation, action-boundary detection, and coarse-to-fine multiple-resolution modeling.The cited approaches target temporal reasoning, over-segmentation, or improved temporal resolution.
  • Recent Methods: A single convolutional layer does not connect all input-output position pairs, leaving room for improvement in temporal modeling.

3 Methods

ASFormer adapts an encoder-decoder Transformer for framewise action prediction and iterative refinement. Its temporal convolutions, hierarchical attention, cross-attention decoders, and combined losses target small datasets, long videos, and prediction smoothness.

  • Overall Architecture: ASFormer encodes pre-extracted frame-wise features into initial action probabilities, then passes predictions through successive decoders for incremental refinement.The architecture uses one encoder and multiple decoder stages.
  • Encoder: The encoder maps T × D feature sequences to predictions y_e ∈ R^T×C, where T is video length, D feature dimension, and C action classes.A fully connected layer first adjusts the input dimension before serial encoder blocks.
  • Encoder: Each encoder block combines dilated temporal convolution, single-head self-attention, residual connections, instance normalization, and ReLU activation.Temporal convolution replaces the vanilla Transformer's point-wise fully connected feed-forward layer to provide local inductive bias.
  • Hierarchical Representation: A predefined hierarchical pattern makes lower self-attention layers focus locally before higher layers capture longer dependencies.The same hierarchical principle is applied to decoder cross-attention, while convolution dilation increases with encoder depth.
  • Hierarchical Representation: The hierarchical pattern reduces approximate space complexity from J · T · T for vanilla Transformers to (2−ε)·2^J · T.With J = 9, 2^J = 512 is described as almost 10 times smaller than T.
  • Decoders: Decoder cross-attention combines encoder and previous-layer information, and multiple decoders perform iterative refinement with weighted residual connections.The external-information weight α is set to 1 for the first decoder and exponentially decreased afterward.
  • Loss Function: Training minimizes summed encoder and decoder losses combining framewise cross-entropy classification with mean-squared smoothness loss.The balance weight λ is set to 0.25 in the experiments.

4 Dataset

The evaluation uses 50Salads, GTEA, and Breakfast, covering kitchen and daily activities with different dataset sizes, action counts, subjects, and cross-validation protocols.

  • Datasets: 50Salads contains 50 videos, 17 action classes, approximately 20 action instances per video, and videos averaging 6.4 minutes.Evaluation uses 5-fold cross-validation and reports average results.
  • Datasets: GTEA contains 28 kitchen-activity videos with 11 action classes, approximately 20 action instances per video, and videos averaging about half a minute.The standard four subject-held-out train-test splits are averaged.
  • Datasets: Breakfast is the largest and most challenging dataset, with 1712 videos recorded in 18 kitchens and 48 action classes.The videos depict breakfast-preparation activities.
  • Evaluation Setup: All three datasets use 2048-dimensional I3D features pre-extracted for each frame.The features come from an I3D model trained on Kinetics.
  • Evaluation Setup: Evaluation reports frame-wise accuracy, segmental edit score, and an additional metric identified in the experimental setup.

5 Experiments

Experiments examine position encoding, attention heads, local temporal convolutions, hierarchical attention, decoder depth, block count, and benchmark performance. ASFormer achieves state-of-the-art results across three datasets while supporting long sequences and serving as a backbone for ASRF.

  • 5.1 Impact of position encoding and multi-head self-attention: Removing position encoding is beneficial for ASFormer, while adding it to the encoder or both encoder and decoders reduces performance on 50Salads.The authors attribute this to temporal convolution modeling relative positional relationships, making redundant absolute encoding potentially harmful.
  • 5.1 Impact of position encoding and multi-head self-attention: Single-head self-attention achieves comparable performance to multiple heads on 50Salads, so ASFormer uses it by default to reduce computation and memory costs.The comparison keeps each multi-head dimension equal to the single-head dimension.
  • 5.2 Effect of the local connective inductive bias: Temporal convolution substantially outperforms an MLP feed-forward layer in the encoder, especially on F1 and Edit scores, showing the importance of neighboring local information.The ablation compares encoder-only variants on 50Salads while avoiding decoder-refinement effects.
  • 5.3 Effect of the hierarchical representation pattern: The non-hierarchical attention version drops significantly, whereas the hierarchical pattern focuses attention on meaningful locations instead of learning near-uniform or nonlocal weights.Figure 2 visualizes normalized attention weights for an anchor frame across encoder blocks.
  • 5.4 Effect of the multiple decoders: Stacking three decoders produces the best refinement result, with decoders substantially improving performance over the encoder alone.The study evaluates different numbers of decoders on 50Salads and visualizes iterative predictions in the encoder and decoders.
  • 5.5 Ablations of the number of blocks: Performance increases with block count through J = 9; J = 10 slightly improves frame-wise accuracy but not F1 scores while sharply increasing GPU memory cost.ASFormer therefore uses J = 9 in all experiments.
  • 5.6 Comparison with SOTA: ASFormer achieves state-of-the-art results on 50Salads, GTEA, and Breakfast, with the evaluated metrics indicating accurate and smooth predictions.The comparison is reported in Table 7 against previous work.
  • 5.6 Comparison with SOTA: Replacing MS-TCN with ASFormer in ASRF yields higher results on 50Salads than the original ASRF, supporting ASFormer as an adaptable backbone.The resulting model is denoted ASRF*.

6 Conclusion

ASFormer explores Transformer-based models for action segmentation and addresses three identified concerns with corresponding design solutions. Results on three public datasets demonstrate the method’s effectiveness.

  • ASFormer explores applying Transformer-based models to the action segmentation task.
  • The method proposes three major concerns and corresponding solutions for action segmentation.
  • Results on three public datasets demonstrate ASFormer’s effectiveness and provide a new solution for the action segmentation community.
Loading 2110.08568v1…