PaperlayerView exact revision on arXiv

Source-linked AI summary

TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks

Humam Alwassel, Silvio Giancola, Bernard Ghanem

arXiv:2011.11479v3cs.CV

TL;DR

Untrimmed-video localization usually relies on clip features from trimmed-action encoders, whose temporal sensitivity is not guaranteed. TSP adds foreground/background and global-video supervision during pretraining, and the resulting features improve localization methods across three tasks and multiple architectures, datasets, and algorithms.

  • Problem

    Localization methods use precomputed clip features because untrimmed videos are impractical to encode directly, but trimmed-action pretraining may not distinguish actions from nearby temporal background.

  • Method

    TSP pretrains video encoders with foreground action labels, foreground/background temporal-region labels, and global video information from untrimmed videos.

  • Results

    TSP features significantly improve state-of-the-art methods on Temporal Action Localization, Action Proposal Generation, and Dense Video Captioning across encoder architectures, pretraining datasets, and localization algorithms.

  • Takeaways & Limitations

    Temporally-sensitive clip features provide a broadly transferable feature foundation for more accurate video localization models.

Abstract

from arXiv · show

Due to the large memory footprint of untrimmed videos, current state-of-the-art video localization methods operate atop precomputed video clip features. These features are extracted from video encoders typically trained for trimmed action classification tasks, making such features not necessarily suitable for temporal localization. In this work, we propose a novel supervised pretraining paradigm for clip features that not only trains to classify activities but also considers background clips and global video information to improve temporal sensitivity. Extensive experiments show that using features trained with our novel pretraining strategy significantly improves the performance of recent state-of-the-art methods on three tasks: Temporal Action Localization, Action Proposal Generation, and Dense Video Captioning. We also show that our pretraining approach is effective across three encoder architectures and two pretraining datasets. We believe video feature encoding is an important building block for localization algorithms, and extracting temporally-sensitive features should be of paramount importance in building more accurate models. The code and pretrained models are available on our project website.

1. Introduction

Video localization methods commonly rely on clip features pretrained for trimmed action classification, which may not distinguish actions from nearby temporal background. TSP addresses this mismatch by incorporating foreground/background supervision and global video information, improving localization across tasks and settings.

  • Motivation: Untrimmed videos are difficult to encode end-to-end on commodity hardware, so most localization methods use precomputed clip features.Some end-to-end methods require aggressive spatial and temporal downsampling.
  • Motivation: Features for Temporal Action Localization, Action Proposal Generation, and Dense Video Captioning are typically extracted from encoders pretrained for Trimmed Action Classification.The paper argues that this representation is not necessarily suitable for localization tasks.
  • TSP: TSP pretrains encoders to classify foreground activities while also using background clips and global video information to improve temporal sensitivity.The approach explicitly targets the distinction between action content and temporal context.
  • Results: TSP features improve state-of-the-art methods on Temporal Action Localization, Action Proposal Generation, and Dense Video Captioning.The reported gains are consistent across encoder architectures, pretraining datasets, and localization algorithms.
  • Results: TSP particularly improves localization of short action instances and helps algorithms draw sharper contrast between foreground and background context.The paper also reports that the features encode background clips differently from foreground clips.

2. Related Work

Prior work spans action recognition, temporal action localization, proposal generation, and dense video captioning. These localization systems commonly depend on video features pretrained for action classification, including in dense captioning pipelines.

  • Action recognition: Action recognition research uses large-scale datasets and architectures such as two-stream encoders and Temporal Segment Networks to model appearance, motion, and temporal information.These encoders are commonly pretrained for trimmed action classification.
  • Temporal localization: Temporal action localization and proposal generation include fully supervised, weakly supervised, single-frame supervised, and self-supervised approaches.Earlier systems also used complex action classifiers in sliding-window pipelines.
  • Dense video captioning: Dense Video Captioning jointly localizes events and describes them textually, with later methods using attentive fusion, transformers, reinforcement learning, and multimodal signals.The task was introduced with the ActivityNet Captions benchmark.
  • Shared feature setting: Dense Video Captioning algorithms, like other localization systems, rely on temporally insensitive TAC-pretrained features that perform worse than TSP-pretrained features.The comparison motivates replacing the conventional feature pretraining strategy.

3. Technical Approach

TSP trains clip encoders on untrimmed videos using action and temporal-region supervision, combining local clip features with a pooled global video feature. Its loss applies both heads to foreground clips and the temporal-region head to background clips, while balancing foreground and background sampling.

  • Traditional pretraining: TAC-pretrained encoders often learn from foreground clips only, limiting their ability to distinguish actions from nearby background context.Localization methods therefore use features that may confuse action instances with temporal context.
  • Input data: TSP samples fixed-size clips from temporally annotated untrimmed videos and assigns action-class and binary temporal-region labels.The temporal-region label marks foreground/action clips with yr = 1 and background/no-action clips with yr = 0.
  • Feature encoding: The encoder produces local clip features, while max-pooling features from all clips produces a global video feature used to condition foreground-versus-background classification.The global feature supplies video-level context that a short clip alone lacks.
  • Classification heads: An action-label head classifies local features, and a temporal-region head classifies the concatenated local and global features.The two heads use separate fully connected layers for action classes and the two temporal regions.
  • Optimization: The loss includes both classification terms for foreground clips but only the temporal-region term for background clips.The coefficients αc and αr weight the action and region losses.
  • Optimization: Training subsamples clips so foreground and background examples contribute equally, while the global video feature is frozen and precomputed from a Kinetics-pretrained encoder.These choices address temporal imbalance and the memory cost of processing whole untrimmed videos.

4. Experiments

Experiments evaluate TSP across localization tasks, encoder architectures, algorithms, and pretraining datasets. TSP features consistently improve localization performance, with gains in temporal precision and transferability, and achieve state-of-the-art or competitive results across tasks.

  • Experimental Setup: Experiments evaluate TSP features on TAL, Action Proposal Generation, and Dense Video Captioning using ActivityNet and THUMOS14.The study compares TSP against TAC on Kinetics, ActivityNet, and THUMOS14, using multiple state-of-the-art localization algorithms.
  • Effects of TSP on Target Tasks: TSP improves average mAP by at least +1.73% on TAL, AUC by +0.96% on Proposals, and average METEOR by +0.54% on Dense-Captioning over both baselines.These comparisons use ActivityNet pretraining with R(2+1)D-34 features and G-TAD, BMN, and BMT, respectively.
  • Effects of TSP on Target Tasks: 37.12% mAP@0.75 for TSP versus 34.87% for TAC on ActivityNet indicates stronger performance at high temporal-overlap thresholds.The study interprets this high-tIoU improvement as better boundary contrast between actions and nearby background.
  • TSP with Different Video Encoders: TSP outperforms TAC across ResNet3D-18, R(2+1)D-18, and R(2+1)D-34, with the performance gap widening for deeper encoders.The gap over TAC on ActivityNet increases from +0.84% for R(2+1)D-18 to +1.73% for R(2+1)D-34.
  • TSP with Other Localization Algorithms: With BMN, TSP maintains at least a 0.92% average-mAP advantage over TAC-based pretraining and reduces the original BMN–G-TAD gap from 0.24% to 0.14%.This shows that TSP features benefit multiple localization algorithms rather than only G-TAD.
  • State-of-the-Art Comparison: TSP reaches 35.81% average mAP on ActivityNet TAL and 53.5% mAP@0.5 on THUMOS14, while improving Proposals and Dense-Captioning over their baselines.It achieves state-of-the-art TAL results on both datasets, reaches 69.04% AUC for Proposals, and obtains 8.75% average METEOR for Dense-Captioning.

5. Feature Analysis

TSP features are analyzed for temporal sensitivity through activity-length comparisons and within-video similarity. They better distinguish foreground activity from surrounding background and improve localization of shorter instances.

  • DETAD analysis: TSP features outperform TAC features on extra-short and short ActivityNet action instances, which are especially difficult to localize.Extra-short instances comprise 53.7% of annotated instances and contain limited temporal and recognition information.
  • Feature similarity: TSP improves temporal sensitivity by making clip features more discriminative between foreground activities and background clips.The analysis expects within-activity consensus and sharp differences from temporally surrounding background.
  • Feature similarity: Figure 2 compares cosine-similarity matrices for clips from individual videos using TAC on Kinetics and TSP features, with green lines marking ground-truth actions.The matrices are shown for multiple videos in columns, with TAC on top and TSP below.
  • Feature similarity: TAC features can make foreground clips resemble background or split one activity, whereas TSP homogenizes activity features and separates them from background.TSP also increases dissimilarity within background regions instead of forcing all background clips to be similar.

6. Conclusion

The paper presents TSP as supervised pretraining that combines action classification with background and global-video information to improve temporal sensitivity. TSP features improve state-of-the-art methods across three localization tasks.

  • Conclusion: TSP combines action classification with background clips and global video information to produce temporally sensitive video-encoder features.The approach targets temporal action localization, action proposal generation, and dense video captioning.
  • Conclusion: TSP features improve state-of-the-art methods on Temporal Action Localization, Action Proposal Generation, and Dense Video Captioning.The conclusion presents these gains across all three target tasks.

A. Pooling Function for GVF

The study compares average-pooled and max-pooled global video features within TSP. Max-pooled global video features perform better across all three target tasks.

  • Pooling function: Max-pooled global video features outperform average-pooled features across Temporal Action Localization, Action Proposal Generation, and Dense Video Captioning.The comparison uses TSP features with R(2+1)D-34 encoders pretrained on ActivityNet and task-specific downstream methods.

B. Extended Ablation Study Results

Extended statistical analyses report mean and standard deviation results for TSP and TAC-pretrained baselines across ablation studies. The reported improvements remain consistent and outside the noise range.

  • Statistical analysis: Extended results show that TSP improvements are consistent with the main-paper findings and do not lie within the noise range.The analysis reports mean and standard deviation performances across five-run ablation studies and all tIoU thresholds.

C. Extended State-of-the-Art Comparison

The extended comparison reports additional methods and all tIoU values across four localization evaluations, with extra ground-truth-proposal results for Dense-Captioning.

  • Tables 15–18 extend SOTA comparisons for TAL on ActivityNet and THUMOS14, Dense-Captioning on ActivityNet Captions, and Proposals on ActivityNet.
  • The extended TAL evaluations include all tIoU thresholds for ActivityNet and THUMOS14.
  • Dense-Captioning additionally reports results using ground-truth proposals.

D. Extended Feature Analysis Study

The extended analyses compare TSP with TAC features, assess robustness across experiments and configurations, and report SOTA performance across localization tasks.

  • Feature Analysis: TSP shows better temporal sensitivity than TAC on Kinetics and better background–foreground distinction than TAC on ActivityNet.Figure 3 compares similarity matrices for the two TAC variants and TSP, with green lines marking ground-truth action extents.
  • Experimental Studies: Extended studies report repeated five-run experiments using mean, standard deviation, and maximum performance values.The studies cover TSP effects, different video encoders, other localization algorithms, and different pretraining datasets.
  • State-of-the-Art Comparison: TSP achieves SOTA performance for TAL on ActivityNet and THUMOS14, using G-TAD and P-GCN respectively.
  • State-of-the-Art Comparison: For Dense-Captioning, TSP achieves SOTA average BLEU and remains competitive in average METEOR.
  • State-of-the-Art Comparison: For Proposals, TSP significantly improves over BMN’s original performance and is competitive with SOTA.
Loading 2011.11479v3…