PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoCapsuleNet: A Simplified Network for Action Detection

Kevin Duarte, Yogesh S Rawat, Mubarak Shah

arXiv:1805.08162v1cs.CV

TL;DR

Action detection remains difficult because existing systems use complex pipelines and scaling capsule routing to videos is computationally expensive. VideoCapsuleNet addresses both issues with a 3D capsule network that jointly classifies actions and produces pixel-wise localization. It achieves state-of-the-art performance across several action-detection datasets, including 20% higher v-mAP on UCF-101 and 15% higher v-mAP on J-HMDB.

  • Problem

    Action detection requires classification and spatial-temporal localization, while existing methods use complex pipelines and video-scale capsule routing is computationally expensive.

  • Method

    VideoCapsuleNet generalizes capsule networks to 3D videos, uses capsule pooling to make routing tractable, and jointly predicts action classes and pixel-wise localization end-to-end.

  • Results

    The network achieves state-of-the-art action-localization performance across multiple datasets, including 20% higher v-mAP on UCF-101 and 15% higher v-mAP on J-HMDB.

  • Takeaways & Limitations

    Predicted capsules capture action representations and support both action classification and pixel-wise localization within one end-to-end network.

Abstract

from arXiv · show

The recent advances in Deep Convolutional Neural Networks (DCNNs) have shown extremely good results for video human action classification, however, action detection is still a challenging problem. The current action detection approaches follow a complex pipeline which involves multiple tasks such as tube proposals, optical flow, and tube classification. In this work, we present a more elegant solution for action detection based on the recently developed capsule network. We propose a 3D capsule network for videos, called VideoCapsuleNet: a unified network for action detection which can jointly perform pixel-wise action segmentation along with action classification. The proposed network is a generalization of capsule network from 2D to 3D, which takes a sequence of video frames as input. The 3D generalization drastically increases the number of capsules in the network, making capsule routing computationally expensive. We introduce capsule-pooling in the convolutional capsule layer to address this issue which makes the voting algorithm tractable. The routing-by-agreement in the network inherently models the action representations and various action characteristics are captured by the predicted capsules. This inspired us to utilize the capsules for action localization and the class-specific capsules predicted by the network are used to determine a pixel-wise localization of actions. The localization is further improved by parameterized skip connections with the convolutional capsule layers and the network is trained end-to-end with a classification as well as localization loss. The proposed network achieves sate-of-the-art performance on multiple action detection datasets including UCF-Sports, J-HMDB, and UCF-101 (24 classes) with an impressive ~20% improvement on UCF-101 and ~15% improvement on J-HMDB in terms of v-mAP scores.

1 Introduction

Action detection requires both recognizing actions and localizing them in space and time, but existing methods often use complex multi-stage pipelines. VideoCapsuleNet proposes a simpler capsule-based network that jointly performs classification and pixel-wise localization.

  • Action detection must identify human actions in long videos and localize them spatially and temporally.
  • Existing approaches commonly extract action proposals, classify regions, and regress bounding boxes in multi-stage pipelines.
  • Optical-flow-based two-stream methods improve performance but require additional optical-flow computation for each video.
  • VideoCapsuleNet generalizes capsule networks to videos, using 3D convolutions and capsules to capture visual and motion characteristics for action recognition.
  • The network uses capsule representations for pixel-wise action localization and is trained end-to-end with classification and localization outputs.
  • ∼15-20% improvement in v-mAP is reported on J-HMDB and UCF-101, alongside state-of-the-art localization results across multiple datasets.

2 Related Work

Prior action-detection systems combine frame or video features with proposals, optical flow, and tube classification. Capsule networks provide an alternative representation based on entity properties and routing-by-agreement, while capsule pooling is introduced to make the approach computationally feasible.

  • CNN-based action methods detect actions frame by frame or from video features, then assemble detections into spatio-temporal tubes.
  • Two-stream CNNs process RGB frames and optical-flow frames, improving motion modeling while requiring separate optical-flow computation.
  • Action-detection methods use proposals such as anchor cuboids or boxes that are linked, classified, and refined through regression.
  • Capsules represent entities with vectors whose orientations encode properties and lengths encode entity existence, with routing measuring agreement between capsule vectors.
  • VideoCapsuleNet extends capsule processing from images to videos and introduces capsule pooling to reduce redundant routing computations.

3 Generalizing capsules to higher dimensional inputs

Scaling capsule networks from images to videos makes routing computationally expensive because high-dimensional inputs create many capsules and votes. VideoCapsuleNet addresses this with shared transformations and capsule pooling over local receptive fields.

  • 3D video inputs create high-dimensional capsule representations, making capsule transformation and routing computationally expensive.
  • Routing between layers requires NL×NL+1 votes, so it becomes intractable when either layer contains too many capsules.
  • Convolutional capsule routing restricts votes to local receptive fields, but large 3D receptive fields can still leave routing expensive.
  • Capsule-Pooling: Transformation matrices are shared among capsules of the same type because they model the same entity at different positions.
  • Capsule-Pooling: Capsule pooling averages capsules of each type within a receptive field before voting and routing-by-agreement.
  • Capsule-Pooling: Pooling reduces routing votes to CL×CL+1, making the count independent of receptive-field volume.

4 Network Architecture

VideoCapsuleNet processes eight video frames with 3D convolutions, convolutional capsule layers, and class capsules. A decoder uses masked class capsules and skip connections to produce pixel-wise localization, while training combines classification and localization losses.

  • The network takes 8 112 × 112 video frames and uses six 3D convolutional layers to produce 512 feature maps of size 8 × 28 × 28.
  • The encoder forms two convolutional capsule layers before connecting the final layer to one class capsule per action class.
  • The class capsule with the largest activation supplies the network’s action prediction.
  • During training, only the ground-truth class capsule is retained; at test time, only the highest-activation predicted class capsule is retained for localization.
  • Objective Function: VideoCapsuleNet is trained end-to-end with a classification loss and a localization loss.
  • Objective Function: The localization prediction has temporal, height, and width dimensions and is trained from pixel-level actionness scores derived from ground-truth bounding boxes.
  • Objective Function: The combined objective is L = Lc + λLs, with λ = 0.0002 to down-weight localization loss.

5 Experiments

Experiments evaluate VideoCapsuleNet on three action-localization datasets, ablate architectural and loss choices, and probe what its class capsules encode. The results show strong localization performance, benefits from coordinate addition, and capsule representations that vary systematically with video motion.

  • 5.1 Results: VideoCapsuleNet achieves a 15% v-mAP improvement on J-HMDB at α = 0.2, while UCF-Sports improves by approximately 1%.At larger f-mAP or v-mAP IoU thresholds, it does not outperform the state of the art, which the authors attribute to the small number of training videos per class.
  • 5.1 Results: 20% higher v-mAP than the most state-of-the-art methods is achieved on the 24-class UCF-101 localization benchmark.The experiments use 3,207 videos with bounding-box annotations.
  • 5.2 What class capsules learn?: Class-specific capsule masking produces action-specific localization information, with ground-truth class masking used for the VideoCapsuleNet* results.The VideoCapsuleNet* row uses ground-truth labels to generate localization maps and is not directly comparable with other state-of-the-art results.
  • 5.3 Ablation Experiments: Adding reconstruction improves performance by 10% when localization information is unavailable, but combining reconstruction and localization losses decreases classification performance.The authors attribute the decrease to learning non-semantic RGB information instead of exploiting semantic bounding-box annotations.
  • 5.3 Ablation Experiments: Coordinate addition improves UCF-101 classification accuracy by about 7% and localization accuracy by about 5%.It adds time, row, and column coordinates to the vote matrices, enabling class capsules to encode positional information.
  • 5.4 Synthetic Dataset Experiments: Synthetic-video experiments show capsule pose dimensions change predictably and smoothly with motion properties, supporting their encoding of spatio-temporal actor characteristics.The learned representations are linked to localization because the capsules represent properties needed for accurate action localization.

6 Conclusion and Future Work

VideoCapsuleNet generalizes capsule networks to 3D videos for joint action classification and pixel-wise localization. Capsule-pooling makes routing feasible, and the network achieves state-of-the-art localization across multiple datasets.

  • VideoCapsuleNet extends capsule networks from 2D images to 3D videos for action classification and pixel-wise localization.
  • Capsule-pooling reduces the computational cost of voting in convolutional capsule layers, making routing feasible.
  • The localization component uses predicted class-specific capsules to generate pixel-wise action localizations.
  • The network is trained end-to-end and achieves state-of-the-art performance on multiple action detection datasets.

Appendix A Synthetic dataset experiments

Synthetic experiments vary controlled motion and appearance properties to examine what VideoCapsuleNet encodes in capsule pose matrices. The pose dimensions change smoothly with several video properties, indicating that the capsules encode action instantiation parameters.

  • The synthetic dataset contains four motion-based action classes with controlled variation in shape, size, color, speed, direction, noise, rotation, and zoom.VideoCapsuleNet is trained on about 200,000 generated videos and evaluated using a 2,000-video hold-out set.
  • The analysis compares average pose-matrix dimensions from randomly generated linear-motion videos with dimensions from videos having specific controlled properties.
  • Capsule pose matrices encode speed, direction, size, and rotation, with most dimensions changing smoothly as these properties vary.This pattern indicates that the capsules encode the instantiation parameters of the actor or video.

Appendix B Localization Results at different thresholds

VideoCapsuleNet outperforms state-of-the-art networks at small v-mAP IoU thresholds across the evaluated datasets, but its performance is less consistent on smaller datasets at higher thresholds. UCF-101 results remain strong across nearly all thresholds.

  • VideoCapsuleNet outperforms state-of-the-art networks on all three datasets when the v-mAP IoU threshold is small.
  • At v-mAP IoU thresholds above 0.4, VideoCapsuleNet slightly under-performs on the smaller UCF-Sports and J-HMDB datasets.The authors attribute this to limited training data, with 10 videos per class in UCF-Sports and 30 videos per class in J-HMDB.
  • On UCF-101, which has about 100 training videos per class, VideoCapsuleNet achieves outstanding results at nearly all IoU thresholds.

Appendix C Class-wise Localizations

Class-wise localization quality varies across actions and datasets. Poor localization is associated with challenging backgrounds or multiple people, while the predicted segmentations can become more form-fitting when pixel-level annotations are available.

  • J-HMDB localization is poor for the Push and Jump actions, whose videos contain very different backgrounds.
  • UCF-101 performance is worst for BasketballDunk and VolleyballSpiking, where multiple humans can be incorrectly classified as foreground actors.

Appendix D More Qualitative Results

VideoCapsuleNet produces box-like localizations on UCF-101 and more form-fitting segmentations on pixel-annotated J-HMDB. Its localizations can sometimes follow actor limbs, but thin arms remain difficult to capture at the input resolution.

  • On UCF-101, VideoCapsuleNet produces box-like action segmentations despite having only bounding-box annotations.Some predictions contour better to the actor’s limbs even without pixel-level action segmentations.
  • On J-HMDB, box-like segmentations become more form-fitting after fine-tuning with pixel-level annotations.The network still often localizes larger body regions, such as the torso and legs, rather than perfectly capturing the actor’s arms.
  • The 112x112 frame size makes thin arms only a few pixels thick, limiting their localization quality.In the baby-clapping example, the localization width adjusts as the hands move closer together.

Appendix E Localization Failure Cases

VideoCapsuleNet fails mainly in crowded scenes and scenes it does not understand, where it may label multiple people or background regions as foreground. These localization errors are often accompanied by misclassification.

  • In videos containing multiple humans but only one foreground actor, the network may label several humans as foreground.This is identified as a common failure case on UCF-101 and J-HMDB test data.
  • The network may also label the actor together with large portions of the background as foreground.This error is usually accompanied by a misclassification, indicating difficulty understanding those scenes.
Loading 1805.08162v1…