PaperlayerView exact revision on arXiv

Source-linked AI summary

Long Short-Term Transformer for Online Action Detection

Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, Stefano Soatto

arXiv:2107.03377v3cs.CV

TL;DR

Online action detection requires causal framewise recognition while modeling prolonged video history without losing fine temporal detail. LSTR separates long- and short-term memories in an encoder-decoder architecture, and experiments establish state-of-the-art performance across three benchmarks. Its reported limitations include runtime dependence on motion extraction and reduced accuracy for tiny motion or distant subjects.

  • Problem

    Online action detection must recognize actions causally in prolonged videos, while existing long-range approaches can lose time-step information or face computational and memory demands.

  • Method

    LSTR uses FIFO long- and short-term memories, compresses long-term features with an encoder, and uses short-term features as decoder queries for temporal attention.

  • Results

    LSTR achieves state-of-the-art online action detection performance on THUMOS’14, TVSeries, and HACS Segment.

  • Takeaways & Limitations

    Experiments and ablations validate separately modeling and integrating long- and short-term information for online inference on prolonged video sequences.

  • Takeaways & Limitations

    Accuracy decreases for actions with tiny motion or distant subjects, while motion feature extraction accounts for about 90% of total runtime.

Abstract

from arXiv · show

We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged sequence data. It consists of an LSTR encoder that dynamically leverages coarse-scale historical information from an extended temporal window (e.g., 2048 frames spanning of up to 8 minutes), together with an LSTR decoder that focuses on a short time window (e.g., 32 frames spanning 8 seconds) to model the fine-scale characteristics of the data. Compared to prior work, LSTR provides an effective and efficient method to model long videos with fewer heuristics, which is validated by extensive empirical analysis. LSTR achieves state-of-the-art performance on three standard online action detection benchmarks, THUMOS'14, TVSeries, and HACS Segment. Code has been made available at: https://xumingze0308.github.io/projects/lstr

1 Introduction

Online action detection must classify each incoming frame causally, while capturing temporal relations across prolonged sequences and retaining fine-grained event information. LSTR addresses this with separate long- and short-term memories in an encoder-decoder architecture, achieving state-of-the-art results on standard benchmarks.

  • Online action detection classifies each frame using only past and current observations, unlike offline methods that access the entire video.
  • LSTR jointly models long- and short-term temporal dependencies to capture both prolonged sequence context and fine-grained event characteristics.
  • LSTR stores history directly, allowing attention to useful frames without recurrent backpropagation through time.
  • Its encoder-decoder architecture compresses long-term memory into fixed-length latent representations while the decoder attends over transient short-term frames.
  • LSTR achieves state-of-the-art online action detection performance on THUMOS’14, TVSeries, and HACS Segment, with ablations supporting its long-sequence modeling ability.

2 Related Work

Related work frames online action detection as causal recognition, distinguishes direct history modeling from finite-state summaries, and notes that Transformer methods often remain limited to short clips.

  • Online action detection processes video causally up to the present, contrasting with temporal localization methods that observe the entire video.
  • Prior online methods use reinforcement learning for early recognition, while TRN combines online detection with anticipation to model greater temporal context.
  • For complex data, finite-dimensional latent states cannot generally summarize all past history, motivating methods that directly model historical observations.
  • Transformer video methods achieve promising temporal-modeling results, but computational and memory demands usually restrict them to short clips.

3 Long Short-Term Transformer

LSTR performs causal online action detection by separating recent fine-grained information from compressed long-term history. Its encoder-decoder design uses Transformer decoder units to efficiently encode extended temporal context and decode current action predictions.

  • Online Inference: During inference, LSTR processes each incoming frame causally, using only current and past observations.The system classifies the action at the latest frame from a temporal stream and does not access future context.
  • Encoder-Decoder Architecture: The encoder compresses long-term memory into a fixed-length latent representation, while the decoder uses short-term frames to retrieve context and predict actions.The decoder treats encoded long-term outputs as input tokens and short-term features as output tokens or queries, producing probability vectors for the short-term frames.
  • Long- and Short-Term Memories: LSTR stores recent and distant frame features in separate short-term and long-term FIFO memories.The short-term memory holds mS recent frames, while older frames enter the mL-slot long-term memory; a typical configuration uses mL = 2048 and mS = 32.
  • LSTR Encoder: Transformer decoder units reduce an m × C input to n × C output tokens in O(n^2C + nmC) time, which becomes linear in m when n ≪ m.Each unit first self-attends over learnable output tokens, then applies cross-attention to the input tokens.
  • Two-Stage Memory Compression: Two-stage memory compression further reduces the cost of encoding long-term memory and empirically improves online action-detection performance.The first stage compresses the entire mL-frame memory to n0 tokens, and subsequent encoder layers operate on the smaller representation.
  • Training LSTR: LSTR trains without temporal unrolling or BPTT and applies directional masking so every short-term frame can receive supervision without future information.Training samples are formed by sampling an ending time and tracing back through the long- and short-term memories; the complete loss sums predictions across short-term frames.

4 Experiments

Experiments evaluate LSTR on three public online action-detection datasets using established feature-extraction and per-frame evaluation protocols.

  • Datasets: Experiments use THUMOS’14, TVSeries, and HACS Segment, covering sports, everyday actions, and large-scale web videos.THUMOS’14 has 20 annotated actions; TVSeries has 30 actions across 16 hours; HACS Segment contains 200 action classes.
  • Feature Encoding: Frames are extracted at 24 FPS, with decisions and accuracy evaluated at 0.25-second intervals.The video chunk size is six frames, and decisions are made at the chunk level.
  • Feature Encoding: Feature encoding combines a ResNet-50 visual model with a BN-Inception motion model applied to stacked optical flow fields.The visual model uses each chunk’s central frame, while the motion model uses flow across six consecutive frames.
  • Implementation Details: The implementation uses PyTorch, 16-head Transformer units with 1024-dimensional hidden states, and Adam optimization with weight decay.Training uses a warmup followed by cosine learning-rate reduction on a system with eight Nvidia V100 GPUs.
  • Evaluation Protocols: Performance is evaluated with per-frame mean average precision and calibrated average precision, with cAP correcting positive-negative sample imbalance.The calibrated metric uses a negative-to-positive ratio in its precision calculation.

4.3 Comparison with the State-of-the-art Methods

LSTR is compared with state-of-the-art methods across online detection and anticipation benchmarks, showing gains on THUMOS’14, TVSeries, and HACS Segment.

  • Online Action Detection: LSTR outperforms existing methods at every TVSeries action stage, especially near action-instance boundaries.The stage analysis evaluates successive ten-percent intervals of action instances.
  • Online Action Detection: 3.7% and 2.4% mAP gains on THUMOS’14 exceed state-of-the-art methods with ActivityNet and Kinetics pretrained features, respectively.The comparison includes methods based on 3D ConvNets, RNNs, reinforcement learning, and curriculum learning.
  • Online Action Detection: 2.8% and 2.7% cAP gains on TVSeries exceed state-of-the-art methods with ActivityNet and Kinetics pretrained features, respectively.LSTR also outperforms existing methods at every evaluated action stage.
  • Online Action Detection: 82.6% mAP on HACS Segment exceeds RNN’s 77.6% by 5.0% and TRN’s 78.9% by 3.7%.This result uses Kinetics pretrained features.
  • Action Anticipation: 7.5% mAP on THUMOS and 2.1% cAP on TVSeries are reported gains for action anticipation up to 2 seconds into the future.The extension predicts eight future steps at 4 FPS using ActivityNet pretrained features.

4.4 Design Choices of Long- and Short-Term Memories

The experiments show that longer long-term memory generally benefits LSTR, while short-term memory helps most when long-term context is limited. Adaptive compression preserves accuracy better than aggressive heuristic downsampling, and recurrent memory partly compensates for short histories.

  • Memory lengths: Longer long-term memory benefits LSTR in most tested cases, while the maximum tested length covers 98% of videos in each dataset.The maximum is 1024 seconds for THUMOS’14 and 256 seconds for HACS Segment.
  • Memory lengths: When long-term memory is ≤16 seconds, larger short-term memory improves results; when it is ≥32 seconds, increasing short-term memory is not consistently beneficial.
  • Downsampling long-term memory: Strides smaller than 4 do not reduce performance, whereas more aggressive downsampling dramatically decreases detection accuracy.
  • Downsampling long-term memory: 69.5% mAP versus 65.9% shows LSTR outperforming stride-128 downsampling when both compress long-term memory to 16 features.
  • Recurrent compensation: Adding a GRU improves performance when long-term memory is small, addressing cases where total accessible memory is limited to 16 seconds or less.

4.5 Design Choices of LSTR

LSTR’s encoder-decoder design is favored over unified self-attention, direct long-memory feeding, alternative encoder designs, and decoder removal. Its two-stage, cross-attention compression and memory integration are also consistently stronger than simpler alternatives.

  • Temporal architecture: 69.5% versus 66.5% mAP at mL = 512 and 66.6% versus 65.7% at mL = 8 show LSTR outperforming a unified Transformer Encoder.
  • Temporal architecture: LSTR outperforms direct long-memory feeding, especially when long-term memory is large, because short-term memory queries related long-range context.
  • Encoder design: LSTR outperforms a standard Transformer Encoder across all tested long-memory settings, while that baseline declines as long-term memory grows.
  • Memory compression: Two-stage compression is stably better than one-stage compression, with the gap increasing from 0.5% at mL = 8 to 0.8% at mL = 512.
  • Decoder contribution: Removing the decoder lowers performance by about 4% for large mL and about 2.5% for relatively small mL.
  • Memory integration: Cross-attention integration stably outperforms heuristic average-pooling and concatenation methods.

4.6 Runtime

LSTR’s two-stage compression is faster than its one-stage alternative, but recurrent models run faster because they maintain more compact history representations. LSTR nevertheless achieves higher reported detection performance, while end-to-end speed is dominated by motion-feature extraction.

  • Design runtime: 91.6 FPS for two-stage compression versus 59.5 FPS for one-stage compression demonstrates the efficiency of the proposed design.
  • Comparison with recurrent models: TRN runs at 123.3 FPS, faster than LSTR, because recurrent models abstract the entire history into a compact representation.
  • Comparison with recurrent models: LSTR outperforms TRN by about 7.5% mAP on THUMOS’14 and about 4.5% cAP on TVSeries.
  • End-to-end runtime: 4.6 FPS is the reported end-to-end online-inference speed, with motion-feature extraction accounting for about 90% of total runtime.

4.7 Error Analysis

Error analysis shows that LSTR is strongest on actions with long temporal span and multiple stages, but accuracy decreases for tiny motions and subjects far from the camera.

  • Failure cases: The illustrated THUMOS’14 failure cases cover BaseballPitch, FrisbeeCatch, Billiards, and CricketShot, with red circles marking the action location.
  • Strengths: LSTR excels at recognizing actions with long temporal span and multiple stages, including PoleVault and Long Jump.
  • Failure cases: Detection accuracy decreases when the action has only tiny motion or the subject is very far from the camera.

5 Conclusion

LSTR captures long- and short-term temporal correlations by compressing long-term memory into latent features and referencing them with short-term memory. Experiments and ablations validate its effectiveness and efficiency for prolonged video sequences, while spatio-temporal extension remains open.

  • LSTR compresses long-term memory into encoded latent features and references related temporal context using short-term memory.
  • Experiments on multiple datasets and ablation studies validate LSTR’s effectiveness and efficiency for prolonged video sequences.
  • LSTR operates only on the temporal dimension, leaving simultaneous spatial and temporal modeling as an open challenge.

7 Appendix

The appendix presents qualitative results, failure cases, and architectural extensions of LSTR, including an extra GRU for additional history and anticipation tokens for future action prediction.

  • Qualitative results and failure cases: LSTR usually recognizes actions quickly and produces relatively consistent predictions, but accuracy decreases for tiny-motion actions such as Billiards.
  • Qualitative results and failure cases: Figure 6 presents qualitative predictions and two typical failure cases, with curves showing predicted scores for ground-truth and background classes.
  • Architectural extensions: Figure 7 depicts LSTR augmented with an extra GRU, while Figure 8 depicts its architecture for action anticipation.
  • LSTR with an extra GRU: An extra GRU captures history outside the long- and short-term memories as a compact representation g ∈R1×C.
  • LSTR with an extra GRU: The LSTR encoder and GRU outputs are concatenated into temporal features of size (n1 + 1) × C before entering the decoder as input tokens.
  • LSTR for action anticipation: For action anticipation up to 2 seconds ahead, mF anticipation tokens are appended after short-term memory tokens and combined with positional embeddings and a directional attention mask.
Loading 2107.03377v3…