PaperlayerView exact revision on arXiv

Source-linked AI summary

MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection

Rui Dai, Srijan Das, Kumara Kahatapitiya, Michael S. Ryoo, Francois Bremond

arXiv:2112.03902v2cs.CV

TL;DR

Densely labelled untrimmed videos contain concurrent and composite actions with complex short- and long-term temporal relations, making action detection challenging. MS-TCT combines convolution and self-attention across temporal scales and predicts instance-center-relative position. The paper reports new state-of-the-art results on three densely labelled benchmarks.

  • Problem

    Densely labelled untrimmed videos can contain concurrent actions with different temporal spans, so detecting their precise temporal boundaries requires modeling short- and long-term dependencies.

    Source: §1 · p. 1

  • Method

    MS-TCT combines convolution and self-attention across temporal scales, fuses multi-scale features, and predicts class-wise relative position to action-instance centers.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 5

  • Results

    MS-TCT achieves new state-of-the-art results on three challenging densely labelled action detection benchmarks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The paper concludes that a ConvTransformer can model local and global temporal relations at multiple scales while learning action-instance center-relative positions.

    Source: §5 · p. 9

    The Visual Encoder and Temporal Encoder are not jointly optimized due to hardware limitations; the authors identify end-to-end modeling of temporal and spatial relations in long untrimmed videos as future work.

    Source: A5. Method Limitation · p. 13

Abstract

from arXiv · show

Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos. The temporal relation is complex in those datasets, including challenges like composite action, and co-occurring action. For detecting actions in those complex videos, efficiently capturing both short-term and long-term temporal information in the video is critical. To this end, we propose a novel ConvTransformer network for action detection. This network comprises three main components: (1) Temporal Encoder module extensively explores global and local temporal relations at multiple temporal resolutions. (2) Temporal Scale Mixer module effectively fuses the multi-scale features to have a unified feature representation. (3) Classification module is used to learn the instance center-relative position and predict the frame-level classification scores. The extensive experiments on multiple datasets, including Charades, TSU and MultiTHUMOS, confirm the effectiveness of our proposed method. Our network outperforms the state-of-the-art methods on all three datasets.

1. Introduction

Densely labelled untrimmed videos contain overlapping actions and short- and long-term dependencies, making precise temporal boundaries difficult to detect. MS-TCT combines convolution and self-attention across temporal scales and adds a branch for instance-center-relative position; the paper reports improved state of the art on three datasets.

1. Introduction

  • Densely labelled untrimmed videos can contain concurrent actions with different temporal spans and limited background information.For example, ‘taking food’ can draw short-term context from ‘opening fridge’ and long-term context from ‘making sandwich’.Source: §1 · p. 1, Figure 1 · p. 1
  • Temporal convolutions access local information, while self-attention can connect distant segments; neither alone readily handles all relevant temporal dependencies.The cited discussion notes that pure transformers must learn adjacent-token relations that temporal convolutions capture through local inductive bias.Source: §1 · pp. 1–2
  • MS-TCT uses convolution and self-attention at multiple temporal scales, then fuses the resulting features into a unified representation.Its heat-map branch predicts class-wise relative position to an action-instance center alongside multi-label classification.Source: §1 · p. 2, Figure 2 · p. 2
  • The work introduces a ConvTransformer for complex temporal relations and a branch for learning instance-center-relative position.Source: §1 · p. 2
  • MS-TCT is reported to improve the state of the art on three challenging densely labelled action datasets.Source: §1 · p. 2

2. Related Work

Prior approaches to densely labelled action detection include temporal filters, convolutions, and transformer-based models, each with stated constraints. MS-TCT is presented as a ConvTransformer that combines global and local temporal modeling across scales.

2. Related Work

  • Temporal filters and convolutions are described as limited in modeling long-range relationships, while coarse-fine networks face input-frame and stride constraints.The discussion also notes limitations of anchor-based methods for dense action distributions and of several approaches to handling composite actions or untrimmed videos.Source: §2 · p. 2
  • MS-TCT combines a Visual Encoder, Temporal Encoder, Temporal Scale Mixer, and Classification Module for action detection.The figure caption identifies the temporal convolution layer as a 1D convolution with kernel size k.Source: Figure 3 · p. 3
  • Transformer-based action detection methods have stated limits for dense labels, long videos, or complex labels.RTD-Net is described as designed for sparsely annotated videos, while MLAD is said to struggle with complex labels.Source: §2 · p. 3

3. Multi-Scale Temporal ConvTransformer

MS-TCT combines temporal modeling at multiple resolutions with a scale mixer and classification branches to detect actions in densely labelled videos. Its temporal encoder alternates global and local relations, while its classification module learns instance-center-relative position alongside frame-level class scores.

3. Multi-Scale Temporal ConvTransformer

  • MS-TCT comprises a Visual Encoder, hierarchical Temporal Encoder, Temporal Scale Mixer, and Classification Module for predicting action probabilities.The Temporal Encoder models relations at different temporal scales; the mixer combines these representations before classification.Source: §3 · p. 3

3.1. Visual Encoder

  • Each 8-frame video segment is encoded by I3D as a temporal token, and the tokens are stacked as a T × D0 representation for the Temporal Encoder.The segments are non-overlapping during training.Source: §3.1 · p. 3

3.2. Temporal Encoder

  • The Temporal Encoder hierarchically reduces token count while increasing feature dimension, alternating global self-attention for long-term relations with temporal convolution for local context.Its stages contain a Temporal Merging Block and B Global-Local Relational Blocks; the first stage uses stride 1, while later merging generally uses stride 2.Source: §3.2 · p. 4

3.3. Temporal Scale Mixer

  • The Temporal Scale Mixer upsamples stage outputs to a common temporal length, adds each earlier-stage representation to a projected last-stage representation, and concatenates the refined scales.The paper reports that this simple fusion performs best, despite the possibility of using more complicated fusion methods.Source: §3.3 · p. 5

3.4. Classification Module

  • The classification module jointly trains a heat-map branch for instance-center-relative positioning and a multi-label branch; only the latter's scores are used in evaluation.The heat-map targets use instance-centered Gaussian responses, and the two branches share the same video features.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate MS-TCT on Charades, TSU, and MultiTHUMOS, densely labelled datasets with overlapping and composite actions. The study reports component ablations, comparisons with prior methods, and analyses of the model's design choices.

4.1. Ablation Study

  • The Temporal Encoder improves the I3D-feature baseline by +7.0%, and the Temporal Scale Mixer adds +0.5% with minimal computational increase.The full model improves the baseline by +9.8%; the heat-map branch works when optimized with the classification branch, but not alone (25.4% vs 10.7%).Source: §4.1 · p. 6, Table 1 · p. 6
  • Removing any Temporal Encoder stage component drops performance, supporting the joint use of global and local relations in its multi-scale structure.Table 2 evaluates single-stage design choices on Charades per-frame mAP.Source: §4.1 · p. 6, Table 2 · p. 6
  • Without temporal convolution, Local Relational Block performance drops; a transition layer adds +1.8% when feature size remains constant.Higher feature expansion also lets temporal convolution better model local relations.Source: §4.1 · p. 7, Table 3 · p. 7

4.2. Comparison to the State-of-the-Art

  • MS-TCT consistently outperforms prior methods on all three evaluated datasets.The comparison uses per-frame mAP on RGB video; Table 4 presents the results.Source: §4.2 · p. 7, Table 4 · p. 7
  • MS-TCT uses only 1/7 of MLAD's FLOPs and exceeds it on all Charades action-conditional metrics; its predictions are more precise than PDAN's.Figure 6 visualizes the ground truth and detections for PDAN and MS-TCT.Source: §4.2 · p. 7, Figure 6 · p. 7, Table 5 · p. 8

4.3. Discussion and Analysis

  • On Charades, ConvTransformer exceeds the pure transformer (22.3%) by +3.1% and the pure convolution model (21.4%) by +4.0%.The comparison uses networks with the same number of stages and similar settings.Source: §4.3 · p. 8, Table 6 · p. 8
  • +0.5%, +0.7%, and +1.3% improvements over MS-TCT without the heat-map branch correspond to three σ settings; the branch improves PDAN by +0.4%.The authors attribute MS-TCT's greater benefit to center-relative cues entering all stages and informing later attention blocks; Figure 7 maps lighter regions to instance centers.Source: §4.3 · p. 8, Table 7 · p. 8, Figure 7 · p. 8
  • Adding learnable temporal positional embeddings lowers Charades performance by 0.2%, which the authors interpret as redundant with the model's implicit temporal positioning.Source: §4.3 · p. 8

5. Conclusion

MS-TCT combines convolutions and self-attention to model local and global temporal relations at multiple scales, and adds a branch for class-wise positions relative to action-instance centers. It achieves new state-of-the-art results on three densely labelled action-detection benchmarks.

5. Conclusion

  • MS-TCT achieves new state-of-the-art results on three challenging densely labelled action-detection benchmarks.The benchmarks are Charades, TSU, and MultiTHUMOS.Source: §5 · p. 9
  • MS-TCT uses convolutions and self-attention to model local and global temporal relations at multiple temporal scales.The model also introduces a branch for learning class-wise positions relative to action-instance centers.Source: §5 · p. 9

Appendix

The appendix previews further experiments on temporal relations, blurred videos, the heat-map branch, and hyperparameters, alongside limitations and Temporal Encoder details.

Appendix

  • The appendix covers analyses of temporal relations, blurred videos, the heat-map branch, and hyperparameters, plus method limitations and further Temporal Encoder details.Source: Appendix · p. 12

A1. Analysis of Temporal Action Relations

On Charades RGB, the ConvTransformer performs better on both short- and long-term temporal dependencies than either a pure convolutional or pure transformer stage.

A1. Analysis of Temporal Action Relations

  • The ConvTransformer performs better on both short- and long-term temporal dependencies than pure convolution or pure transformer stages.The analysis uses action-conditional mAP on Charades RGB; pure convolution is better for τ = 5, while the transformer-based model is better for τ = 100.Source: A1. Analysis of Temporal Action… · p. 12, Table 8 · p. 12

A2. Performance on Blurred Videos

On TSU, where subjects’ faces are blurred for privacy, MS-TCT outperforms state-of-the-art methods, indicating its action detection does not rely on person-identity information.

A2. Performance on Blurred Videos

  • On TSU, MS-TCT outperforms state-of-the-art methods despite blurred faces, indicating it does not rely on person-identity information for action detection.Faces are blurred on TSU to protect personal privacy.Source: A2. Performance on Blurred Videos · p. 12

A3. More studies on Heat-map Branch

The heat-map branch benefits from features at multiple stages, with the all-stage configuration performing best. It also makes predictions more continuous, reflecting tokens’ instance-center-relative positions.

A3. More studies on Heat-map Branch

A4. More Studies on Hyper-parameters

The Charades hyperparameter studies balance model performance against computation and model size. They support an eight-head, three-block design, a temporal-convolution kernel size of 3, 256 tokens, and four encoder stages.

A4. More Studies on Hyper-parameters

  • On Charades, MS-TCT sets the number of attention heads to 8 to balance channels per head with relations modeled; changing head count leaves FLOPs unchanged, while more heads model more complex relationships.FLOPs increase with the number of Global-Local Relational Blocks.Source: A4. More Studies on Hyper-parameters · p. 12
  • Removing the Local Relational Block’s temporal convolution significantly reduces performance, while kernel-size changes make little difference; the chosen size is 3.The study selects 3 because larger kernels add weight parameters.Source: A4. More Studies on Hyper-parameters · p. 13, Table 12 · p. 13
  • 25.0%, 25.4%, and 25.5% on Charades correspond to 128, 256, and 512 tokens, respectively.Performance changes little across these token counts, but FLOPs increase with token count; 256 tokens (2048 frames, about 86 seconds) is the chosen trade-off.Source: A4. More Studies on Hyper-parameters · p. 13
  • A 4-stage Temporal Encoder strikes a balance between performance and model size.The number of stages determines the level of semantic information in the representations.Source: A4. More Studies on Hyper-parameters · p. 13, Table 13 · p. 13

A5. Method Limitation

MS-TCT’s performance remains relatively low, and hardware limitations prevent joint optimization of its Visual and Temporal Encoders.

A5. Method Limitation

  • MS-TCT remains below 30% on Charades, and hardware limitations mean its Visual and Temporal Encoders are not jointly optimized.The proposed future direction is end-to-end modeling of temporal and spatial relations for long untrimmed videos.Source: A5. Method Limitation · p. 13

A6. Which actions benefit the most?

The analysis finds that MS-TCT’s performance varies with action-class characteristics, while detecting both short and long instances consistently. The accompanying architecture description details the Temporal Encoder’s feature sizes and stage-specific settings.

A6. Which actions benefit the most?

A7. Temporal Encoder Architecture

  • Table 14 describes Temporal Encoder input/output feature sizes and stage-specific hyperparameters; each stage contains three repeated Global-Local Relational Blocks.The architecture discussion reports no further improvements from allocating different hyperparameters across stages.Source: A7. Temporal Encoder Architecture · p. 13, Table 14 · p. 14

Related papers

Loading 2112.03902v2…