PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classification

Yuxin Peng, Yunzhen Zhao, Junchao Zhang

arXiv:1711.03273v1cs.CV

TL;DR

Video classification research has often modeled static and motion information separately and learned spatial or temporal attention in isolation. TCLSTA jointly learns spatial-temporal attention and collaborates across static and motion streams with adaptive fusion. The paper reports the best performance among more than 10 state-of-the-art methods across four datasets.

  • Problem

    Many deep video methods model static and motion information separately, and existing approaches also learn spatial or temporal attention in isolation despite their coexistence in video.

    Source: §I · pp. 1–2

  • Method

    TCLSTA jointly learns spatial-temporal attention and uses collaborative guidance with adaptively learned fusion weights for static and motion streams.

    Source: §I · p. 2

  • Results

    TCLSTA achieves the best performance among more than 10 state-of-the-art methods across four datasets; on HMDB51, it exceeds the highest compared result by 2.3%.

    Source: §I · p. 2, C. Comparisons with state-of-the-art… · p. 9

  • Takeaways & Limitations

    The approach combines attention across salient regions and discriminative frames with mutual guidance between complementary static and motion features.

    Source: §V · p. 12

Abstract

from arXiv · show

Video classification is highly important with wide applications, such as video search and intelligent surveillance. Video naturally consists of static and motion information, which can be represented by frame and optical flow. Recently, researchers generally adopt the deep networks to capture the static and motion information \textbf{\emph{separately}}, which mainly has two limitations: (1) Ignoring the coexistence relationship between spatial and temporal attention, while they should be jointly modelled as the spatial and temporal evolutions of video, thus discriminative video features can be extracted.(2) Ignoring the strong complementarity between static and motion information coexisted in video, while they should be collaboratively learned to boost each other. For addressing the above two limitations, this paper proposes the approach of two-stream collaborative learning with spatial-temporal attention (TCLSTA), which consists of two models: (1) Spatial-temporal attention model: The spatial-level attention emphasizes the salient regions in frame, and the temporal-level attention exploits the discriminative frames in video. They are jointly learned and mutually boosted to learn the discriminative static and motion features for better classification performance. (2) Static-motion collaborative model: It not only achieves mutual guidance on static and motion information to boost the feature learning, but also adaptively learns the fusion weights of static and motion streams, so as to exploit the strong complementarity between static and motion information to promote video classification. Experiments on 4 widely-used datasets show that our TCLSTA approach achieves the best performance compared with more than 10 state-of-the-art methods.

I. INTRODUCTION

The introduction argues that video classification needs to model both spatial and temporal attention jointly and learn static and motion information collaboratively. It presents TCLSTA as a two-model approach to these gaps and reports strong results across four datasets.

I. INTRODUCTION

  • Video classification is challenging because categories can contain varied objects and contexts, while different categories can share similar content.The introduction gives birthday-party videos and soccer actions as examples.Source: §I · p. 1
  • Deep two-stream methods often learn static and motion information separately, despite their complementary clues for representing the same video category.Frames provide static information, while optical flow represents motion information.Source: §I · p. 1
  • Existing methods often learn spatial-level or temporal-level attention separately, although salient regions and discriminative frames vary together over a video sequence.The paper argues that attention at each level can guide the other toward discriminative objects and frames.Source: §I · pp. 1–2
  • TCLSTA jointly models spatial-temporal attention and collaboratively learns static and motion features, with adaptive fusion weights for video classification.The attention model emphasizes salient frame regions and discriminative frames; the collaborative model mutually guides the static and motion streams.Source: §I · p. 2

II. RELATED WORKS

  • The related-works section reviews video classification through feature representation and visual attention, and also covers alternate training schemes.It frames alternate training as relevant to the proposed approach.Source: §II · p. 2

1) Methods based on hand-crafted features:

The related-work discussion surveys hand-crafted and deep video representations, then reviews attention methods. It positions TCLSTA's collaborative learning as a way to exploit the complementarity of static and motion features.

1) Methods based on hand-crafted features:

  • Hand-crafted video representations include low-, mid-, and high-level features, spanning local visual patterns, action parts, intermediate concepts, and semantic concepts.Examples include HOG for static appearance, HOF and MBH for motion, and trajectories for combined representation.Source: 1) Methods based on hand-crafted… · p. 3

2) Methods based on deep features:

  • Deep-feature methods are grouped into spatial-temporal networks, multiple-stream networks, deep generative networks, and temporal-coherency networks.The survey describes 3D CNNs and LSTMs among spatial-temporal approaches, and frame/optical-flow inputs among multiple-stream approaches.Source: 2) Methods based on deep features · p. 3
  • Static and motion information are complementary: appearance can distinguish broad action categories, while motion can distinguish actions within a category.The paper uses basketball dunk versus basketball shooting as an example and proposes collaborative learning with optimized frame and optical-flow fusion weights.Source: 2) Methods based on deep features · p. 4

1) Spatial-level attention:

  • Spatial-level attention methods assign different attention to frame regions, but salient regions are not always centered.The cited approaches include selecting regions with an RNN and applying soft attention within CNN layers.Source: 1) Spatial-level attention · p. 4

2) Temporal-level attention:

TCLSTA jointly models spatial and temporal attention, then collaboratively learns from frame and optical-flow features. Its attention streams emphasize salient regions and discriminative frames, while alternate training supports mutual guidance between static and motion features.

2) Temporal-level attention:

  • Earlier temporal-attention methods selected discriminative frames using fixed thresholds or boosted frame selection, while TCLSTA jointly learns spatial and temporal attention.Source: 2) Temporal-level attention · p. 4

C. Alternate training scheme

  • Alternate training optimizes one task while fixing the other, and TCLSTA applies it to enable mutual guidance between static and motion features.Source: C. Alternate training scheme · p. 4

III. OUR TCLSTA APPROACH

  • TCLSTA decomposes videos into frames and optical flow, jointly models spatial-temporal attention, then optimizes features and learns category-specific fusion weights.Source: §III · p. 4

A. Spatial-temporal attention model

1) Connection network:

  • The connection network uses a weighted pooling layer to link spatial attention to feature learning and connects its output to the temporal-attention network for joint training.Source: 1) Connection network · p. 5

2) Spatial-level attention network:

3) Temporal-level attention network:

The temporal-attention network uses LSTM contextual features to assess frame relevance across a video sequence. Its attention weights the spatially attended frame features, yielding features with both spatial and temporal attention information.

3) Temporal-level attention network:

B. Static-motion collaborative model

The static-motion collaborative model alternately uses each stream to guide optimization of the other. This is intended to exploit complementary cues and improve the learned representations.

1) Collaborative learning network:

B. Static-motion collaborative model

1) Collaborative learning network:

2) Adaptively weighted learning:

Static and motion streams contribute differently across categories, so TCLSTA learns a separate fusion weight for each category. It combines stream scores using those weights to produce the final classification.

2) Adaptively weighted learning:

IV. EXPERIMENTS

Experiments evaluate TCLSTA on three trimmed video datasets and one untrimmed dataset, comparing it with more than 10 state-of-the-art methods.

IV. EXPERIMENTS

  • TCLSTA is evaluated on HMDB51, UCF50, UCF101, and THUMOS14 against more than 10 state-of-the-art methods.The first three are trimmed datasets; THUMOS14 is untrimmed.Source: §IV · p. 7

A. Datasets

The experiments use three trimmed action-video datasets and the large-scale untrimmed THUMOS14 dataset, with dataset-specific evaluation protocols.

A. Datasets

  • HMDB51 has 6,766 videos across 51 action categories and three train-test splits, while UCF50 contains 6,618 YouTube videos in 50 categories.UCF50 videos are divided into 25 groups that may share people, backgrounds, or viewpoints.Source: A. Datasets • · p. 7
  • UCF101 contains 13,320 YouTube clips across 101 categories, and evaluation follows the common three-split train-test setting.The clips total over 27 hours and have a fixed frame rate of 25 FPS.Source: A. Datasets • · p. 7
  • THUMOS14 has 101 categories and includes long untrimmed videos; training uses its training and validation sets, with evaluation on the test set.UCF50 uses leave-one-group-out evaluation over 25 groups; HMDB51 and UCF101 average accuracy over three splits, while THUMOS14 reports mean average precision.Source: A. Datasets • · pp. 7–8

B. Implementation details

The implementation uses frame and optical-flow streams with spatial and temporal attention, then collaboratively learns static and motion features. Across four datasets, TCLSTA outperforms the compared methods, with reported gains on the trimmed benchmarks and improved performance on untrimmed THUMOS14.

B. Implementation details

  • Both streams use the same architecture; optical flow is precomputed with TVL1, and 10 consecutive flow fields are stacked into a 20-channel image.The connection network is based on ResNet-50, pretrained on ImageNet, and produces 2048-dimensional frame and optical-flow features.Source: 1) Spatial-temporal attention model · p. 8

2) Static-motion collaborative model:

  • The static-motion collaborative model alternately optimizes static and motion features and includes adaptively weighted learning, with λ set to 5 × 10^-3 by cross-validation.The collaborative network uses two softmax layers and cross-entropy loss.Source: 2) Static-motion collaborative model · p. 8

C. Comparisons with state-of-the-art methods

D. Performances of components in our TCLSTA approach

The component evaluation is organized around two aspects.

D. Performances of components in our TCLSTA approach

1) Effectiveness of spatial-temporal attention model:

The experiments report gains from spatial and temporal attention, with their combination outperforming either alone. Collaborative learning and category-specific adaptive fusion further improve results across the tested datasets.

1) Effectiveness of spatial-temporal attention model:

2) Effectiveness of collaborative learning model:

  • Adding the collaborative learning network to two-stream STA improves classification accuracy and MAP score over late fusion without the network.The compared Two-stream + STA and Two-stream + STA + CLN results use late fusion.Source: 2) Effectiveness of collaborative… · p. 11, Table VII · p. 11
  • Two-stream + STA + CLN + AWL achieves the best accuracy among the four tested collaborative-model settings.The authors report that AWL also improves accuracy over Two-stream + STA and that CLN and AWL reinforce each other.Source: 2) Effectiveness of collaborative… · p. 11, Table VII · p. 11
  • Category-specific adaptive weighted learning achieves the best accuracies on all four datasets among the compared fusion methods.The authors report relatively lower accuracies for early and late fusion, while MKL is slightly better than those methods; AWL learns a specific fusion weight for each category.Source: Table VIII · p. 11, 2) Effectiveness of collaborative… · p. 12

E. Evaluation on efficiency

On HMDB51, TCLSTA is less efficient than two compared shallow-network methods but reportedly outperforms them on accuracy. The conclusion also identifies attention and static-motion representations as targets for further work.

E. Evaluation on efficiency

  • The reported test-speed comparison uses an Intel i7-5930K PC with 64GB RAM and a TITAN X GPU with 12GB memory.For hand-crafted-feature methods, the authors report local-feature extraction speed for simplicity.Source: E. Evaluation on efficiency · p. 12
  • TCLSTA is slightly less efficient than and but clearly outperforms them on accuracy.The efficiency comparison measures test-time running speed on HMDB51; the compared methods and use relatively shallow networks.Source: E. Evaluation on efficiency · p. 12

V. CONCLUSIONS

  • Across four widely used video-classification datasets, the authors report that TCLSTA achieves the best performance compared with more than 10 state-of-the-art methods.Source: §V · p. 12
  • Future work will pursue better spatial-temporal attention and more discriminative static-motion representations.The authors also propose applying unsupervised learning to use large amounts of unlabeled online video.Source: §V · p. 12

Related papers

Loading 1711.03273v1…