Source-linked AI summary
NUTA: Non-uniform Temporal Aggregation for Action Recognition
Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Hao Chen, Joseph Tighe
TL;DR
Action-recognition methods commonly aggregate temporal information uniformly, although some video segments are more informative than others. NUTA selects informative features non-uniformly and synchronizes them with a uniform branch; the network achieves state-of-the-art or comparable performance across four datasets with less computation cost.
Problem
Many action-recognition methods treat frames equally, while identifying informative temporal parts efficiently and exchanging information across sampling strategies remain challenges.
Source: §1 · pp. 1–2Method
NUTA selects temporal features at the feature level, combining a non-uniform clip-level branch with a uniform local-feature branch through temporal synchronization.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3Results
The NUTA network achieves state-of-the-art or comparable performance with less computation cost on Kinetics 400, Kinetics 700, Something Something V2, and Charades.
Source: §1 · p. 2Takeaways & Limitations
Visualizations show NUTA skipping repetitive, background, or noisy temporal portions while focusing on representative features.
Source: §5 · p. 8
Abstract
from arXiv · showhide
In the world of action recognition research, one primary focus has been on how to construct and train networks to model the spatial-temporal volume of an input video. These methods typically uniformly sample a segment of an input clip (along the temporal dimension). However, not all parts of a video are equally important to determine the action in the clip. In this work, we focus instead on learning where to extract features, so as to focus on the most informative parts of the video. We propose a method called the non-uniform temporal aggregation (NUTA), which aggregates features only from informative temporal segments. We also introduce a synchronization method that allows our NUTA features to be temporally aligned with traditional uniformly sampled video features, so that both local and clip-level features can be combined. Our model has achieved state-of-the-art performance on four widely used large-scale action-recognition datasets (Kinetics400, Kinetics700, Something-something V2 and Charades). In addition, we have created a visualization to illustrate how the proposed NUTA method selects only the most relevant parts of a video clip.
1. Introduction
Action-recognition methods have largely focused on comprehensive spatial-temporal feature extraction, often weighting frames equally, while this work asks where to extract features. NUTA selects informative temporal features and synchronizes a non-uniform branch with a uniform branch for local and clip-level representation.
1. Introduction
- Most prior methods emphasize comprehensive spatial-temporal feature extraction, although frames may differ in importance for recognizing an action.Temporal pooling, striding, and convolution aggregate features uniformly, whereas NUTA extracts non-uniformly from informative parts of a clip.Source: §1 · p. 1, Figure 1 · p. 1
- Selecting informative temporal episodes efficiently is challenging because identifying them requires looking across the video.Separate frame-selection and recognition stages cannot use later feature extraction to improve selection and can add redundancy.Source: §1 · p. 1
- NUTA projects T temporal instances to fewer instances, T′, using feature-level selection end to end rather than selecting frames before recognition.Its temporal receptive field spans the clip, enabling clip-level features.Source: §1 · p. 2
- A two-branch network combines uniform local features with non-uniform clip-level features through temporal synchronization.Synchronization aligns the branches for information exchange; the design is reported to remove noise and retain similar computational cost to a single-branch network.Source: §1 · p. 2
- NUTA achieves state-of-the-art or comparable performance on four action-recognition datasets with less computation cost.The datasets are Kinetics 400, Kinetics 700, Something Something V2, and Charades.Source: §1 · p. 2
2. Related Work
Action-recognition research progressed from frame-based and segment-based methods to 3D CNNs and other approaches emphasizing how to model temporal information. NUTA instead addresses where to extract features, selecting informative temporal episodes end to end rather than adding features or separating frame selection from recognition.
2. Related Work
- Earlier action-recognition approaches progressed from 2D CNNs and recurrent models to segment-level methods and 3D CNNs for spatiotemporal modeling.The related work notes that LSTM methods often increased computation with little accuracy benefit.Source: §2 · p. 2
- Recent methods such as non-local networks, TEA, TAM, SlowFast, and TPN primarily improve how temporal features are modeled or combined.A smaller set of methods targets where to sample, including SCSampler and multi-agent sampling of informative clips.Source: §2 · p. 2
- NUTA differs by selecting features non-uniformly at the feature level end to end, rather than adding redundant information or selecting frames before classification.Its two branches model complementary features from the same input, unlike many two-branch systems that use different modalities or representations.Source: §2 · pp. 2–3
3. Methodology
NUTA pairs uniform local-temporal features with non-uniform clip-level features, synchronizes the branches, and fuses information between them.
3.1. Overview
- The network combines a uniform branch for local spatial-temporal features with a non-uniform branch that samples clip-level temporal features.Information is exchanged between branches after each stage.Source: §3.1 · p. 3
3.2. Non-uniform Temporal Down-sampling
- The learned temporal projection maps T instances to T′ instances, with T′ = T/2 in this paper.A multi-head design lets channels select different temporal periods; group convolution is used for computational efficiency, with groups set to 64.Source: §3.2 · p. 3
- The projection map represents temporal associations based on spatial features, unlike the pixel-level spatial-temporal associations of a non-local kernel.The cited comparison distinguishes the map M from the non-local kernel M′.Source: §3.2 · p. 3
3.3. Temporal Synchronization
- Synchronization applies the same temporal projection to uniform-branch features, aligning them with non-uniform features despite differing temporal dimensions and sampled frames.The following 3D ResNet layers still generate local features; the authors state synchronization removes non-informative features and reduces computation cost.Source: §3.3 · pp. 3–4
3.4. Information Fusion
- At each stage, the method fuses uniform-branch features with the preceding non-uniform features; channel concatenation performs best at the lowest computational cost among the three strategies.The alternatives are non-local fusion, feature sum-up, and channel concatenation.Source: §3.4 · p. 4
3.5. Implementation Details
- Training uses 64 Tesla V100 GPUs across eight machines, with batch size 8 per GPU and 180 training epochs.When batch size is below 8 per GPU, synchronized batch normalization is used.Source: §3.5 · p. 4
- The training setup uses random resized crops, random temporal sampling, horizontal flips, and dropout of 0.6 before the final linear layer.Videos shorter than the required frame count are padded by repeating the last frame.Source: §3.5 · p. 5
4.1. Dataset
The evaluation covers four action-recognition datasets, using top-1 accuracy for three and mAP for Charades.
Kinetics 400
- Kinetics 400 has about 240k training and 20k validation videos across 400 classes; evaluation reports validation top-1 accuracy.Videos are trimmed to 10 seconds.Source: §4.1 · p. 5
Kinetics 700 and Something-Something V2
- Kinetics 700 has about 650k clips across 700 classes, while Something-Something V2 has 220,847 videos across 174 actions; both use validation top-1 accuracy.Kinetics 700 clips last around 10 seconds; Something-Something V2 features object-manipulation actions that require strong temporal modeling.Source: §4.1 · p. 5
Charades
- Charades contains about 9.8k training and 1.8k validation videos in 157 multi-label classes, with performance measured by mAP.Activities average about 30 seconds.Source: §4.1 · p. 5
4.2. Comparison to State-of-the-art
Across four action-recognition datasets, NUTA reports state-of-the-art or comparable performance, often with lower computation. Results include a 63.0% top-1 accuracy on Something-Something V2 and a compute–accuracy trade-off on Charades.
4.2.1 Kinetics 400
- 3.2% higher performance over I3D comes with about 8% more computation; NUTA also compares favorably with Kinetics-400 methods at similar or lower FLOPs.The reported comparisons include 1.2% higher accuracy than non-local at roughly half the computation, and 0.7% higher accuracy than TEA at roughly the same FLOPs using 16 frames.Source: §4.2.1 · pp. 5–6, Table 1 · p. 5
4.2.2 Kinetics 700
- 10% higher performance than baseline I3D on Kinetics-700 accompanies performance comparable to SlowFast; SlowFast used Kinetics-400 and Kinetics-600 pretraining, while NUTA used Kinetics-400.The authors suggest the pretraining difference may explain the small gap between NUTA and SlowFast with non-local.Source: §4.2.2 · p. 6, Table 2 · p. 6
4.2.3 Something-Something V2
- 63.0% top-1 accuracy establishes state-of-the-art single-model RGB performance on Something-Something V2, with an 11.5% boost over the I3D baseline.The model applies NUTA directly to an unmodified I3D backbone.Source: §4.2.3 · p. 6, Table 3 · p. 6
4.2.4 Charades
- 41.2 mAP with 18% fewer FLOPs than SlowFast-101 and SlowFast-101 with non-local reflects a Charades accuracy–computation trade-off; K700-pretrained NUTA reaches state-of-the-art performance at lower computation.Charades is a longer-duration, multi-label action-classification dataset.Source: §4.2.4 · pp. 6–7, Table 4 · p. 7
4.3. Ablation Study
The ablations compare NUTA with uniform temporal aggregation and vary branch use, frame rate, resolution, convolution groups, and NUTA stages. They report gains and trade-offs, while showing that adding stages does not always improve performance.
4.3.1 NUTA vs. uniform temporal aggregation
- 1.5% additional performance over I3D without temporal down-sampling is reported for NUTA, with roughly 10% additional GLOPs.The authors suggest the no-down-sampling I3D retains more temporal information; NUTA modules add limited overhead because they perform their own temporal down-sampling.Source: §4.3.1 · p. 7
4.3.2 NUTA network branch analysis
- 0.7% higher accuracy from adding a NUTA branch while classifying only uniform-branch features is followed by a further 0.9% gain when non-uniform-branch features are included.The authors conjecture that the uniform branch uses the temporal mapping learned by NUTA; they describe the two branches' features as complementary.Source: §4.3.2 · p. 7
4.3.3 Frame rate
- Denser frame sampling gives better performance but costs significantly more computation; it also prevents early information loss and gives NUTA units more input for aggregation.The stated benefit is specific to supplying the NUTA units with sufficient information for temporal feature aggregation.Source: §4.3.3 · p. 7
4.3.4 Frame resolution
- Larger input resolution generally improves performance while significantly increasing computational cost.The paper uses 2562 resolution for its other experiments and GFLOPs estimates.Source: §4.3.4 · p. 7
4.3.5 NUTA configuration
- 64 groups in the NUTA module's 3D convolutions give the best performance among the tested group counts.The comparison varies the number of groups used in the 3D convolutions.Source: §4.3.5 · p. 7
4.3.6 Number of NUTA units
- Adding NUTA stages does not necessarily improve performance; stages 4 and 5 together give the best reported result.The authors suggest that stacking too many modules can over-compress features because each NUTA downsamples the temporal dimension by two.Source: §4.3.6 · p. 7
5. Visualization
The temporal projection visualization shows NUTA aggregating uniformly when information is distributed smoothly and concentrating on informative segments when clips contain irrelevant content or noise.
5. Visualization
- For smooth transitions, NUTA performs uniform temporal sampling, as in the top clip.The paper describes smooth transitions as indicating information is roughly uniformly distributed over time.Source: §5 · pp. 7–8, Figure 3 · p. 8
- When frames are repetitive or contain only background, NUTA skips non-informative frames and focuses on representative features.In the middle example, information from the first half of the clip is skipped.Source: §5 · p. 8, Figure 3 · p. 8
- With scene changes or noise such as transition frames, NUTA focuses on representative information by skipping noise.In the bottom example, it focuses on the start and end of the video while skipping the middle frames.Source: §5 · p. 8, Figure 3 · p. 8
6. Conclusion
The conclusion presents NUTA as a way to select informative features, paired with uniform sampling to learn local features and non-uniform sampling to learn clip-level features.
6. Conclusion
- NUTA selects informative features to address where video features should be extracted.Source: §6 · p. 8
- The two-branch network uses a uniform branch for local features and a non-uniform branch for clip-level features.Source: §6 · p. 8
- The authors report state-of-the-art accuracy on four public action-recognition datasets and identify action detection as a future application.Source: §6 · p. 8