Source-linked AI summary
Appearance-and-Relation Networks for Video Classification
Limin Wang, Wei Li, Wen Li, Luc Van Gool
TL;DR
Video architectures must capture both appearance and temporal relations, but existing designs either require costly optical flow or model these cues jointly and implicitly. The paper introduces ARTNet, which stacks SMART blocks with separate appearance and relation branches, and reports superior performance across Kinetics, UCF101, and HMDB51. Optical flow remains complementary, although its computational cost limits real-world use.
Problem
Video representation learning requires capturing both static appearance within frames and temporal relations across frames, while existing two-stream and 3D-CNN designs have important limitations.
Method
ARTNet stacks SMART blocks that explicitly separate spatial appearance modeling from temporal relation modeling and optimize the resulting multi-scale network end to end.
Results
ARTNet achieves superior performance on Kinetics, UCF101, and HMDB51, including better Kinetics performance than C3D and superior transfer performance to the original C3D.
Takeaways & Limitations
Separately modeling appearance and relation improves spatiotemporal feature learning over 3D convolution under the reported RGB and transfer settings.
Takeaways & Limitations
Optical flow provides complementary information for ARTNet, but its high computational cost limits application in real-world systems.
Abstract
from arXiv · showhide
Spatiotemporal feature learning in videos is a fundamental problem in computer vision. This paper presents a new architecture, termed as Appearance-and-Relation Network (ARTNet), to learn video representation in an end-to-end manner. ARTNets are constructed by stacking multiple generic building blocks, called as SMART, whose goal is to simultaneously model appearance and relation from RGB input in a separate and explicit manner. Specifically, SMART blocks decouple the spatiotemporal learning module into an appearance branch for spatial modeling and a relation branch for temporal modeling. The appearance branch is implemented based on the linear combination of pixels or filter responses in each frame, while the relation branch is designed based on the multiplicative interactions between pixels or filter responses across multiple frames. We perform experiments on three action recognition benchmarks: Kinetics, UCF101, and HMDB51, demonstrating that SMART blocks obtain an evident improvement over 3D convolutions for spatiotemporal feature learning. Under the same training setting, ARTNets achieve superior performance on these three datasets to the existing state-of-the-art methods.
1. Introduction
Video understanding requires modeling both static appearance within frames and temporal relations across frames. ARTNet addresses this by stacking SMART blocks that explicitly separate these functions and achieves improvements across major action-recognition benchmarks.
- Video classification depends on capturing static appearance in individual frames and temporal relations across multiple frames.
- Existing approaches include two-stream CNNs, 3D CNNs, and 2D CNNs combined with temporal models.
- Two-stream CNNs are effective but require training two networks and computing optical flow in advance, while 3D CNN performance remains lower than two-stream CNNs.
- SMART separates appearance and relation modeling into explicit branches, using linear combinations for spatial structure and multiplicative interactions for temporal dynamics.
- ARTNet stacks SMART blocks hierarchically to capture increasingly coarse and long-range spatiotemporal structure, with branch responses concatenated and reduced.
- ARTNet improves over C3D on Kinetics and achieves performance improvements when transferred to HMDB51 and UCF101.
- ARTNet is investigated on Kinetics with RGB-only input and training from scratch, obtaining state-of-the-art performance under that setting.
2. Related Work
Prior video architectures and multiplicative-interaction models provide the context for ARTNet. ARTNet differs by learning supervised relation modeling jointly with an explicit appearance branch in a two-branch architecture.
- Video-classification research has explored two-stream CNNs, 3D CNNs, and 2D CNNs paired with temporal models.
- ARTNet focuses on short-term temporal modeling and differs from 3D CNNs by separating appearance and relation explicitly rather than modeling them jointly and implicitly.
- Multiplicative interactions have been used to model correspondence, image transformations, motion information, and invariant features.
- The SMART relation branch is inspired by earlier multiplicative-interaction work and uses a square-pooling architecture similar to ISA.
- Unlike previous approaches, SMART learns relation-branch weights through supervised backpropagation and integrates the relation branch with an appearance branch.
3. Spatiotemporal Feature Learning
The method separates appearance modeling from temporal relation modeling, using multiplicative interactions to reduce dependence on patch content and SMART blocks to fuse both feature types. ARTNet stacks SMART blocks to learn hierarchical spatiotemporal representations while retaining the design role of 3D convolutions.
- Multiplicative interactions: Standard feature learning on concatenated patches couples appearance with relation, increasing modeling difficulty and over-fitting risk.
- Multiplicative interactions: Multiplicative interactions define transformation codes from quadratic forms of two patches, making activations less dependent on patch appearance.
- Multiplicative interactions: Factorizing the quadratic parameter tensor reduces the roughly cubic parameter growth and connects the formulation to energy models.
- SMART blocks: SMART blocks use separate appearance and relation branches: 2D convolution captures spatial structure, while square pooling models temporal relations.
- SMART blocks: The two branch outputs are concatenated and reduced with a 1×1×1 convolution to form a compact feature volume.
- ARTNet: ARTNet stacks SMART blocks hierarchically, replacing 3D convolutions in C3D-ResNet18 while capturing increasingly longer-range visual structure.
4. Experiments
The experiments evaluate ARTNet and its SMART blocks on Kinetics, including block designs, stacking, input modalities, long-term modeling, and comparisons with prior methods, then test transfer to UCF101 and HMDB51.
- Evaluation settings: Experiments evaluate ARTNet on Kinetics, UCF101, and HMDB51, using RGB-only training from scratch for the main Kinetics studies.UCF101 and HMDB51 evaluations use representation transfer from Kinetics, with accuracy averaged over three splits.
- Building blocks: 77.4% versus 75.7% shows that SMART blocks outperform 3D convolutions on Kinetics, while relation branches reach 77.2%.The comparison uses ResNet18-based building blocks and measures performance on the Kinetics validation set.
- Block stacking: 78.7% versus 77.0% shows that ARTNet-ResNet18 outperforms the deeper C3D-ResNet34 after stacking SMART blocks.The authors attribute the improvement to SMART blocks rather than increased network depth.
- Two-stream inputs: 78.7% versus 78.2% shows that RGB-only ARTNet slightly outperforms two-stream C3D, while two-stream ARTNet reaches 80.4%.Two-stream C3D improves from 75.7% to 78.2%, but the remaining experiments mainly compare RGB input because of two-stream computational cost.
- Long-term modeling and Kinetics comparison: 80.0% versus 77.8% on validation and 78.7% versus 78.2% on test show that ARTNet with TSN slightly outperforms the compared competitive methods.TSN supplies long-term modeling while ARTNet focuses on short-term spatiotemporal feature learning.
- Transfer learning: ARTNet outperforms C3D by 3.7% on UCF101 and 5.5% on HMDB51 after Kinetics pre-training.ARTNet achieves 94.3% on UCF101 and 70.9% on HMDB51 with TSN, and the paper reports comparable performance to the best RGB-3D method trained at a larger scale.
5. Conclusion and Future Work
ARTNet uses SMART blocks to separately and explicitly model appearance and relation, with reported gains over 3D convolution and C3D across RGB and transfer settings.
- SMART models appearance and relation separately using linear combinations within frames and multiplicative interactions across frames.
- ARTNet with a single RGB input outperforms C3D with two-stream inputs on Kinetics.
- ARTNet achieves superior performance to the original C3D when transferring representations from Kinetics to UCF101 and HMDB51.
- Optical flow improves ARTNet performance but remains costly for real-world systems.
- The authors plan to reduce the single-stream versus two-stream performance gap and explore deeper ARTNets at larger spatial resolutions.