Source-linked AI summary
Video Modeling with Correlation Networks
Heng Wang, Du Tran, Lorenzo Torresani, Matt Feiszli
TL;DR
Video action recognition commonly uses optical flow as motion input or 3D convolutions to model appearance and motion together. This paper proposes learnable correlation matching over feature maps to combine explicit motion cues with appearance throughout the network. It reports favorable comparisons with 3D CNNs and competitive results against a prominent two-stream network, with much faster training.
Problem
Action-recognition models use optical flow input or 3D convolutions to capture motion, and the paper explores an alternative that integrates explicit temporal matching with appearance cues.
Source: §1 · p. 1Method
Results
The correlation network compares favorably with widely used 3D CNNs and is competitive with a prominent two-stream network while training much faster.
Source: §1 · p. 1Takeaways & Limitations
Correlation networks produce strong action-recognition results across varied video datasets, while action localization, optical flow, and geometry matching are proposed as future applications.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Motion is a salient cue to recognize actions in video. Modern action recognition models leverage motion information either explicitly by using optical flow as input or implicitly by means of 3D convolutional filters that simultaneously capture appearance and motion information. This paper proposes an alternative approach based on a learnable correlation operator that can be used to establish frame-toframe matches over convolutional feature maps in the different layers of the network. The proposed architecture enables the fusion of this explicit temporal matching information with traditional appearance cues captured by 2D convolution. Our correlation network compares favorably with widely-used 3D CNNs for video modeling, and achieves competitive results over the prominent two-stream network while being much faster to train. We empirically demonstrate that correlation networks produce strong results on a variety of video datasets, and outperform the state of the art on four popular benchmarks for action recognition: Kinetics, Something-Something, Diving48 and Sports1M.
1. Introduction
Video action recognition models capture motion through optical flow input or 3D convolutions, which combine appearance and temporal modeling. This paper proposes learnable feature-map correlation to integrate explicit motion and appearance information throughout a network.
1. Introduction
- Two-stream networks combine RGB appearance features with optical-flow motion features, while 3D convolutions model appearance and motion with spatiotemporal kernels.Two-stream representations are typically fused in a late network layer.Source: §1 · p. 1
- The paper introduces learnable correlation to match convolutional feature maps across frames, integrating explicit motion cues with appearance information throughout the network.Unlike two-stream models that fuse disjointly learned motion and appearance subnets late, the approach integrates them throughout; unlike 3D CNNs, it factorizes their computation.Source: §1 · p. 1
- The correlation network compares favorably with widely used 3D CNNs and achieves competitive results against a prominent two-stream network while training much faster.The experiments cover Kinetics, Something-Something, Diving48, and Sports1M.Source: §1 · p. 1
- The stated contributions include an efficient learnable-filter operator and a network integrating motion and appearance in every block, with state-of-the-art results on four video datasets.The contribution statement also notes that the results are obtained without using optical flow, though the supplied passage ends mid-phrase.Source: §1 · p. 1
2. Related Work
Prior video-recognition work includes two-stream networks and 3D CNNs, as well as methods that learn motion without optical flow and applications of correlation to matching tasks. The paper positions its approach as a learnable correlation operator used repeatedly to model appearance and motion together.
Architectures for video classification
- Two-stream networks [46] [57] [16] [58] [42] and 3D CNNs [53] [50] [55] [63] [9] [14] are the two major video-model categories described.3D CNNs learn appearance and motion simultaneously; decomposing 3D convolution into 2D spatial and 1D temporal convolution has been reported to improve performance [50] [55] [42].Source: §2 · p. 2
Motion information for action recognition
Applications of correlation operation
- Correlation supports dense optical-flow correspondence through hand-crafted patch features and multiplicative patch comparisons, as well as geometric transformation estimation [44] and object tracking [17].In action recognition, [8] defines correlation as relationships among spatial and temporal feature-map dimensions, unlike the adjacent-frame matching studied in this work.Source: §2 · pp. 2–3
- The paper extends prior correlation-based ideas with a learnable operator repeatedly combined with other operators to learn appearance and motion without explicitly or implicitly estimating optical flow.The passage characterizes the resulting architecture as achieving state-of-the-art accuracy on various video datasets.Source: §2 · p. 3
3. Correlation Operator
The proposed operator matches feature patches across images or adjacent video frames, with learned filters and grouping designed to retain useful matching information and channel capacity efficiently. Its video extension preserves sequence length by adding self-correlation for the first frame.
Correlation operator for matching
- The operator computes patch similarity between image feature maps, restricting candidate matches to a K×K neighborhood where K is the maximum displacement.For tractability, feature patches may be reduced to single pixels, making them C-dimensional vectors whose similarity is their dot product.Source: §3 · p. 3
Learnable correlation operator
- Learnable filters weight channel contributions and spatial offsets, allowing matching to select informative channels and represent differing displacement patterns.The filter size is C × K × K; dilation with K = 7 and D = 2 covers a 13 × 13 region without increasing computational cost with K.Source: §3 · p. 3
Groupwise correlation operator
- Because ordinary correlation maps C input channels to K² outputs, repeated use can sharply reduce channel dimension and representation power.The groupwise variant splits channels into G groups, computes within each group, and stacks outputs to produce K²G channels without additional computational cost.Source: §3 · pp. 3–4
From two images to a video clip
- For video, correlation is computed between every adjacent frame pair, with self-correlation on the first frame keeping output length equal to the input length.This extension is intended to make the operator easier to use when designing architectures.Source: §3 · p. 4
Correlation operator and 3D convolution
- The operator targets matching information between adjacent frames, whereas 3D convolution learns spatial and temporal representations together; the correlation operator has much fewer parameters and FLOPs.The comparison in the passage refers to Table 1.Source: §3 · p. 4
4. Correlation Network
The correlation network adds temporal feature matching to an R(2+1)D backbone while retaining appearance information. It uses two block designs and inserts correlation blocks at selected backbone stages.
4. Correlation Network
- Correlation blocks combine temporal matching with appearance operators to provide a comprehensive feature set for video classification.The network inserts them into an R(2+1)D backbone.Source: §4 · p. 4
- The correlation-sum block reduces channels before matching, restores them afterward, and uses a residual shortcut.Its design uses 1×1×1 convolutions around the correlation operator.Source: §4 · p. 4
- The correlation-concat block joins a correlation branch with a 1×1×1-convolution branch along the channel dimension.Figure 2 marks the number of channels for each operator.Source: §4 · p. 5, Figure 2 · p. 5
- Correlation blocks follow res2, res3, and res4, but not res5, whose spatial resolution is 7 × 7.The correlation operator adds only a small computational overhead because its FLOPs are much lower than 3D convolution.Source: §4 · p. 5
5. Experimental Setups
The experiments span four video datasets with differing action-recognition emphases. Training uses RGB clips, with dataset-specific training schedules and evaluation splits.
5. Experimental Setups
- The evaluation covers Kinetics, Something-Something, Diving48, and Sports1M, which vary in scale and action-recognition focus.Something-Something emphasizes actions with everyday objects, while Diving48 is designed to reduce scene and object-context bias and focus on temporal dynamics.Source: §5 · p. 5
- Kinetics and Something-Something are evaluated on validation sets, while Diving48 and Sports1M use the authors’ testing setup.Sports1M videos are cut into shorter clips, producing about 5M training samples.Source: §5 · p. 5
- Training samples contain 16 or 32 RGB frames at 224 × 224 resolution, and testing averages predictions from 10 uniformly spaced clips.Some Something-Something videos with insufficient frames have each frame repeated twice.Source: §5 · p. 5
- Models train from scratch except in Section 6.3, for 150–500 epochs depending on dataset size.Training lasts 250 epochs on Kinetics, 150 on Something-Something and Diving48, and 500 on Sports1M.Source: §5 · p. 5
6. Experimental Evaluation
The evaluation compares CorrNet with baseline backbones and a two-stream network, tests design choices, measures state-of-the-art performance, and visualizes learned filters.
6.1. Correlation network vs baseline backbones
- CorrNet improves over R(2+1)D by over 3% on three datasets, including a 4.3% gain on Diving48 with 16 frames, at a small FLOPs increase.The reported FLOPs rise from 71.9 to 74.8; longer clips improve accuracy across datasets, with Something and Diving48 benefiting more.Source: Table 3 · p. 6, §6.1 · p. 6
6.2. Evaluating design choices and comparison to two-stream network
- Removing filters or grouping drops accuracy by about 1%, correlation-concat underperforms correlation-sum, and CorrNet alone matches the two-stream R(2+1)D model.The two-stream model doubles backbone FLOPs, while optical-flow computation adds further cost.Source: §6.2 · pp. 6–7, Table 4 · p. 6, Table 5 · p. 6
- Accuracy gains from enlarging filter size become marginal beyond K = 7, possibly because feature maps have low resolution.Figure 3 presents the effect of filter size K on classification accuracy.Source: Figure 3 · p. 6, §6.2 · p. 7
6.3. Comparison to the state of the art
- CorrNet-101 reports 51.7% on Something versus 43.9% for MFNet-C101, 38.6% on Diving48 versus 21.4% for R(2+1)D, and 77.1% on Sports1M, 1.6% above ip-CSN-152.The paper also reports 79.2% for CorrNet-101 versus 78.9% for SlowFast-101 on Kinetics.Source: Table 7 · p. 7, §6.3 · pp. 7–8, Table 8 · p. 8
6.4. Visualizing Correlation Filters
- The visualized 32×64×7×7 filters show temporally similar motion patterns within columns and different channel activity across columns.The white arrows mark the directions with the largest weights; some channels receive larger weights, indicating greater discriminativeness for matching.Source: §6.4 · p. 8, Figure 4 · p. 8
7. Conclusions
The paper introduces a learnable correlation operator that explicitly models motion by matching frames across convolutional feature maps, and reports strong action-recognition performance across video datasets.
7. Conclusions
- The learnable correlation operator establishes frame-to-frame matches over convolutional feature maps across network layers to learn motion information.Unlike optical flow or 3D convolution, it makes motion computation explicit.Source: §7 · p. 8
- The correlation network combines the operator with other operators to learn appearance and motion information simultaneously.Source: §7 · p. 8
- The authors report superior performance on various video datasets for action recognition.Source: §7 · p. 8
- Potential future applications include action localization, optical flow, and geometry matching.Source: §7 · p. 8