PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Structured State-Spaces for Long-Form Video Understanding

Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, Raffay Hamid

arXiv:2303.14526v1cs.CV

TL;DR

Long-form video understanding requires modeling complex spatiotemporal dependencies beyond the short clips targeted by many existing methods. The paper introduces S5 for adaptive token selection and LSMCL for robustness, achieving state-of-the-art performance across three challenging benchmarks, with up to 9.6% improvement over ViS4mer on LVU.

  • Problem

    Most existing video-understanding methods focus on short videos, while treating all image tokens equally can hinder S4-based long-term dependency modeling.

  • Method

    S5 uses a lightweight mask generator guided by S4 features to adaptively select informative tokens, while LSMCL trains robustness to mis-predicted tokens and longer temporal context from shorter clips.

  • Results

    The method achieves state-of-the-art performance across LVU, COIN, and Breakfast, including up to 9.6% improvement over ViS4mer on LVU.

  • Takeaways & Limitations

    Adaptive token selection and long-short masked contrastive learning provide an effective approach for long-form video understanding while improving robustness and temporal context.

Abstract

from arXiv · show

Effective modeling of complex spatiotemporal dependencies in long-form videos remains an open problem. The recently proposed Structured State-Space Sequence (S4) model with its linear complexity offers a promising direction in this space. However, we demonstrate that treating all image-tokens equally as done by S4 model can adversely affect its efficiency and accuracy. To address this limitation, we present a novel Selective S4 (i.e., S5) model that employs a lightweight mask generator to adaptively select informative image tokens resulting in more efficient and accurate modeling of long-term spatiotemporal dependencies in videos. Unlike previous mask-based token reduction methods used in transformers, our S5 model avoids the dense self-attention calculation by making use of the guidance of the momentum-updated S4 model. This enables our model to efficiently discard less informative tokens and adapt to various long-form video understanding tasks more effectively. However, as is the case for most token reduction methods, the informative image tokens could be dropped incorrectly. To improve the robustness and the temporal horizon of our model, we propose a novel long-short masked contrastive learning (LSMCL) approach that enables our model to predict longer temporal context using shorter input videos. We present extensive comparative results using three challenging long-form video understanding datasets (LVU, COIN and Breakfast), demonstrating that our approach consistently outperforms the previous state-of-the-art S4 model by up to 9.6% accuracy while reducing its memory footprint by 23%.

1. Introduction

Long-form video understanding requires modeling complex long-term spatiotemporal dependencies, but existing approaches face quadratic transformer costs and task-dependent limitations in uniform S4 token treatment. The paper proposes adaptive token selection with S5 and LSMCL to improve efficiency, robustness, and temporal context.

  • Motivation: Most established video models focus on short clips and are not designed for complex long-form spatiotemporal dependencies.The cited approaches include two-stream, recurrent, and 3-D convolutional networks.
  • Motivation: O(S^2T^2) complexity makes standard video transformers prohibitively expensive for videos with T frames and S spatial tokens.The cost affects both computation and memory when modeling long-form videos.
  • Motivation: Uniform image-token treatment can be suboptimal because S4 models exhibit different temporal reasoning preferences across downstream tasks.The paper motivates task-specific token selection rather than applying one fixed selection strategy to all tasks.
  • Approach: S5 uses a lightweight mask generator guided by S4 features and momentum distillation to adaptively select informative image tokens without dense self-attention.The selected tokens are used by S4 to learn discriminative long-form video representations.
  • Approach: LSMCL pre-training improves tolerance to mis-predicted tokens and enables shorter clips to achieve on-par performance with longer clips without LSMCL.It uses randomly selected tokens from long and short clips and matches their representations.
  • Results: The combined techniques achieve state-of-the-art performance across LVU, COIN, and Breakfast, with up to 9.6% LVU improvement and 23% lower memory footprint than S4.These results are reported as part of the paper’s comparative evaluation of long-form video understanding.

2. Related Work

Related work spans long-form video modeling, adaptive token selection, and video self-supervised learning. The paper positions S5 as addressing video-level long-range reasoning and dense-attention costs while LSMCL addresses risks from incorrectly dropped tokens.

  • Long-Form Video Modeling: Transformer-based video methods model long-term dependencies but face dense self-attention bottlenecks and are not designed for the many spatial and temporal tokens in long-form videos.Efficient transformer variants improve attention costs but remain mismatched to long-form video token structure.
  • Adaptive Token Selection: Adaptive token selection methods use motion, audio, recurrent agents, or transformer modules to filter informative clips, frames, or tokens.Examples include SCsampler, Adaframe, AR-NET, and transformer-based selection methods.
  • Video Self-Supervised Learning: Prior token-reduction methods rarely address mis-dropped tokens, while video SSL methods learn discriminative features using long and short clips.The paper connects this gap to its masked contrastive-learning design.

3. Approach

The approach uses S4’s linear-time sequence modeling as a foundation, then introduces task-adaptive token selection and long-short masked contrastive pretraining for robust long-form video understanding.

  • S4 model: S4 maps input sequences through a latent state and output using learned state-space parameters, addressing long-sequence modeling with structured dynamics.The continuous formulation is discretized for token inputs before producing outputs through the state recurrence.
  • S4 model: S4 computes outputs as a structured convolution over all previous inputs, with computational cost linear in sequence length and efficient FFT-based evaluation.Its convolution kernel is formed from the discretized state-space parameters.
  • ViS4mer model: ViS4mer represents a video as spatial-temporal image tokens with separate spatial and temporal positional encodings before applying S4-based temporal reasoning.Each frame is decomposed into patches, projected into D-dimensional embeddings, and combined with positional encodings.
  • S4 model in long-form video understanding: Increasing input duration does not substantially improve performance, while random masking generally degrades performance and fails to remove redundancy effectively.The analysis compares longer clips with random masking ratios from 20% to 80% on LVU tasks.
  • Adaptive token in long-form videos: S5 uses a lightweight mask generator guided by momentum-updated S4 features to adaptively select discriminative image tokens without dense self-attention.The momentum S4 model is updated as a moving average of the S4 model, and the mask generator samples tokens using a differentiable Gumbel-Softmax procedure.
  • Long-short masked contrastive learning: LSMCL pretrains S5 with independently masked long and short clips so representations remain robust to incorrectly selected tokens while modeling longer temporal context.The long clip contains the short clip temporally, and contrastive learning matches their representations using an S4 backbone and momentum encoder.

4. Experiments

Experiments evaluate S5 on three long-form video benchmarks and show that adaptive token selection improves efficiency and performance, while LSMCL strengthens robustness and temporal modeling.

  • Datasets: S5 is evaluated on LVU, COIN, and Breakfast, covering long-form understanding and procedural activity classification tasks.LVU contains nine tasks; COIN and Breakfast evaluate long-range procedural activities.
  • Efficiency: 25% lower memory usage is achieved by S5 with a linear mask generator while maintaining the same throughput as ViS4mer.More intricate transformer mask generators do not improve memory consumption or throughput.
  • Ablation Study: 0.5% average improvement on LVU is obtained by stacking multiple S5 models, although gains diminish on less redundant sequences.The default configuration therefore uses one S5 block for memory efficiency and throughput.
  • Ablation Study: LSMCL improves S5 robustness to incorrectly selected tokens and captures longer temporal context as the long-short sampling stride ratio increases.Its benefit becomes insignificant above a 50% masking ratio because insufficient information remains.
  • Comparison with the State-Of-The-Arts: 9.6% improvement over ViS4mer is achieved on LVU, while S5 improves over ViS4mer by 2.4% on COIN and 5.5% on Breakfast.The method also outperforms other state-of-the-art methods on these benchmarks.
  • Overall Results: S5 learns discriminative long-form video representations by selecting informative image tokens while combining performance gains with memory efficiency.The experiments support effectiveness across three challenging benchmarks.

5. Conclusion

The paper presents S5, a selective S4 model that adaptively chooses informative video tokens, and LSMCL pretraining to improve robustness and temporal context. Experiments show effectiveness across three long-form video benchmarks.

  • Conclusion: S5 uses a lightweight mask generator and momentum-guided S4 features to adaptively select informative tokens without dense self-attention.The method dynamically learns from informative tokens for different long-form video tasks.
  • Conclusion: LSMCL pretraining mitigates the impact of less informative selected tokens while broadening the model’s temporal horizon.It is designed to improve robustness and use longer temporal context.
  • Conclusion: The proposed components achieve new state-of-the-art performance on three challenging long-form video understanding benchmarks.The conclusion reports extensive experiments supporting both proposed components.

A. Implementation Details

This section introduces additional training information for the S5 model and LSMCL.

  • Implementation Details: Additional training details for S5 and LSMCL are provided beyond the main paper’s implementation section.The supplementary section expands the description of model training.

A.1. S5 model

The implementation uses specified optimization, momentum-update, training-duration, and hardware settings for S5 experiments.

  • S5 model: A momentum coefficient of 0.01 is used for updating the momentum S4 model because it produced the best performance.The experiments explore different momentum-coefficient values.
  • S5 model: S5 models are trained for 100 epochs on COIN, Breakfast, and each LVU task, with learning-rate reduction after one epoch without decreasing training loss.The reported training setup uses task-specific datasets and a plateau-based schedule.
  • S5 model: Training uses 8× NVIDIA Tesla V100 16G GPUs with batch size 16, and implementations are written in PyTorch.These are the reported hardware and software settings.

A.2. LSMCL

LSMCL trains on pairs of clips sampled with different strides, using momentum-encoded query and key pathways to learn longer temporal context from shorter inputs. Its training setup varies across datasets and uses a dedicated contrastive objective.

  • Clip Sampling: LSMCL samples two clips from each video sequence using different sampling strides.The clips use the same shape as S5 fine-tuning inputs.
  • Clip Sampling: Input clips contain 60 frames on LVU and 64 frames on COIN and Breakfast, with each frame represented at 3 × 224 × 224 resolution.These dataset-specific clip sizes are used for LSMCL sampling.
  • Encoder Design: The query encoder combines an S4 backbone, projection head, and prediction head, while the key encoder uses an S4 backbone and projection head.The prediction layer transforms the query representation to match the key representation.
  • Training Setup: LSMCL training runs for 300 epochs with learning-rate warm-up during the first 40 epochs.The reported training setup uses 8 NVIDIA Tesla V100 16G GPUs and batch size 64.
  • Model Variants: The study evaluates replacing additional S4 models with S5 models in the ViS4mer architecture on LVU.Unless otherwise noted, experiments use 60 input frames and a 50% masking ratio.

B. Effectof Multiple S5 Models

The multiple-S5 study follows ViS4mer’s three-stage S4 architecture and examines substituting more of its S4 modules with S5 modules. The default configuration replaces only the first S4 module while retaining the remaining architecture.

  • Architecture: ViS4mer uses three S4 models with pooling and MLP layers between them.The multiple-S5 experiments retain this architecture for fair comparison.
  • Model Variants: The experiments substitute progressively more S4 models in ViS4mer with S5 models.This directly tests the effect of leveraging multiple S5 modules.
  • Default Setting: The default setting replaces the first S4 model with S5 and keeps the remaining S4 models unchanged.The paper notes that S5’s advantages are naturally diminished on less redundant sequences.
Loading 2303.14526v1…