Source-linked AI summary
Long-Term Feature Banks for Detailed Video Understanding
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühl, Ross Girshick
TL;DR
Video models often see only short clips, limiting their access to the past and broader context needed for understanding. The paper adds a time-indexed long-term feature bank to existing 3D CNNs, producing state-of-the-art results across AVA, EPIC-Kitchens, and Charades.
Problem
Most video models base predictions on short 2-5-second clips because densely sampled 3D convolutions must fit within GPU memory, limiting long-term context.
Method
A long-term feature bank stores time-indexed representations from across a video and augments a standard 3D CNN with flexible supportive information.
Results
Augmenting 3D CNNs with a long-term feature bank yields state-of-the-art results on AVA, EPIC-Kitchens, and Charades.
Takeaways & Limitations
Long-term context provides supportive information that improves detailed video understanding across datasets with object-, frame-, and video-level outputs.
Takeaways & Limitations
The feature bank cannot be trained jointly end-to-end because back-propagation through it is computationally and memory intensive; its CNN and detector are fixed after offline training.
Abstract
from arXiv · showhide
To understand the world, we humans constantly need to relate the present to the past, and put events in context. In this paper, we enable existing video models to do the same. We propose a long-term feature bank---supportive information extracted over the entire span of a video---to augment state-of-the-art video models that otherwise would only view short clips of 2-5 seconds. Our experiments demonstrate that augmenting 3D convolutional networks with a long-term feature bank yields state-of-the-art results on three challenging video datasets: AVA, EPIC-Kitchens, and Charades.
1. Introduction
The paper addresses the difficulty of understanding present video content from short clips by augmenting 3D CNNs with a time-indexed long-term feature bank. This supportive context improves recognition across object-, frame-, and video-level tasks, achieving state-of-the-art results on three challenging datasets.
- Human observers continually relate present events to the past, motivating video models that can use longer-term context.
- The long-term feature bank stores rich, time-indexed features from past and, when available, future scenes, objects, and actions.
- Short video models typically process only 2-5 seconds because densely sampled 3D convolutions must fit within GPU memory.
- Unlike prior pre-computed-feature approaches, the bank decouples long-term context from the standard 3D CNN and can store flexible information such as object detections.
- An attentional mechanism can relate present information from a 3D CNN to long-range banked information across object-level, frame-level, and video-level tasks.
- Augmenting 3D CNNs with the bank yields state-of-the-art results on AVA, EPIC-Kitchens, and Charades.
2. Related Work
Prior video models either model short clips, use pre-computed features without end-to-end task optimization, or aggressively subsample video. The proposed approach targets detailed video understanding with flexible long-term context.
- 3D CNNs and two-stream networks dominate video understanding, while temporal models use RNNs or multilayer perceptrons for ordered frame features.
- Fine-grained video methods also model object proposals or detections through co-occurrence, temporal order, or spatial arrangement within short clips.
- Long-term understanding with modern CNNs is constrained by GPU memory, encouraging pre-computed features, aggressive subsampling, or large temporal strides.
- Pre-computed-feature methods avoid end-to-end training for target tasks, while TSN samples 3-7 frames and ST-ResNet uses a temporal stride of 15.
- Unlike short-clip action localization methods that process frames or clips independently, this approach exploits long-term context for detailed video understanding.
- The paper draws inspiration from bank representations used for images, video indexing and retrieval, and text-corpus information modeling.
3. Long-Term Feature Bank Models
The long-term feature bank augments short-clip 3D CNNs with time-indexed features from across a video, enabling present-focused predictions to use distant context. A feature bank operator relates current RoI features to a temporal window of bank features through attention or pooling-based mechanisms.
- Long-Term Feature Bank: The method extends short-clip 3D CNNs with a time-indexed feature bank representing actors and events across the video.The bank stores RoI features computed at regularly sampled time steps and can cover the entire video.
- Feature Bank Operator: The feature bank operator combines current short-term RoI features with a centered temporal window of long-term features.The resulting representation is concatenated with the short-term feature and passed to a linear classifier.
- Operating Modes: The model supports both batch processing with past and future context and causal online processing using only past information.The temporal window size 2w + 1 is cross-validated in experiments.
- Implementation Details: The backbone is an ImageNet-pretrained ResNet-50 inflated into a 3D CNN with non-local blocks and limited temporal downsampling.RoI features are produced by temporal average pooling, RoIAlign, and spatial max pooling.
- Feature Bank Operator: Attention-based and pooling-based operators provide alternative ways to relate present features to long-term context.The default LFB NL operator uses non-local blocks, while LFB Avg/Max uses average or max pooling over the feature window.
- Training: End-to-end training through the long-term bank is infeasible, so the detector and 3D CNN used to compute the bank are fixed after offline target-dataset training.A short-term-only baseline permits back-propagation but exhibits substantial overfitting.
4. Experiments on AVA
The AVA experiments evaluate long-term feature banks for spatio-temporal action localization, testing temporal support, feature design, model complexity, and final benchmark performance. Across these studies, adding long-term context improves 3D CNN-based recognition beyond short-term or ensembled baselines.
- Dataset and setup: AVA contains 235 training and 64 validation movie videos, with sparse 1-FPS person boxes and multilabel annotations for 80 atomic actions.Actions are labeled within ±0.5 seconds of each labeled frame.
- Dataset and setup: The AVA models use 32-frame 3D CNN inputs spanning approximately 2 seconds, while long-term features are computed once per second across each video.The short- and long-term features use the same temporal sampling span, but the bank covers the entire video.
- Temporal Support: Increasing temporal support by striding hurts vanilla 3D CNN and STO, whereas adding LFB improves performance from 22.1 to 25.5.The online causal setting shows a similar trend.
- Temporal Support: Long-range context improves AVA performance despite actions being localized within ±0.5 seconds, motivating a 60-second LFB window for subsequent ablations.The authors focus on the batch setting for the rest of the ablation study.
- Feature-bank design: Non-local feature-bank integration outperforms pooling, while actor-level detected features outperform regular-grid and global pooled representations.These comparisons test both the temporal aggregation function and the spatial granularity of stored features.
- Complexity and benchmark results: The LFB model uses roughly twice the baseline parameters and computation, but baseline ensembles do not obtain a similar gain.With an R101-I3D-NL backbone and standard testing, LFB reaches 27.7 mAP; its RGB-only model exceeds the prior single-model entry by 5.8 validation and 6.2 test mAP points.
5. Experiments on EPIC-Kitchens
On EPIC-Kitchens, the long-term feature bank improves segment-level verb, noun, and action recognition by supplying task-specific context beyond the short clip.
- The feature bank stores different information for the verb and noun tasks, using motion-oriented features for verbs and object-related context for nouns.Verb features are extracted every 1 second; default windows are 40 seconds for verbs and 12 seconds for nouns.
- The EPIC-Kitchens task predicts verbs, nouns, and combined actions from annotated daily-activity segments, evaluated with top-1 and top-5 accuracy.
- 5.7% absolute noun improvement (26.1 →31.8) results from augmenting the strong baseline with object features.
- 3.2% verb improvement (49.8 →53.0) with 3D CNN features also exceeds Baradel et al. by 12.1% (40.9 →53.0).
- FBOMax and FBOAvg outperform FBONL on EPIC-Kitchens, consistent with the dataset’s simpler single-person, single-activity setting.The authors conjecture that the simpler setting makes a simpler function sufficient.
- On test action accuracy, LFB improves over Two-Stream TSN by 50% relatively for seen kitchens and nearly doubles performance for unseen kitchens.
6. Experiments on Charades
On Charades, LFB models consistently outperform models without LFB, although gains are smaller for the coarser video-level prediction task.
- Charades contains 9,848 videos averaging 30 seconds, with video-level recognition of one or more actions without localization.
- A 2× shorter training schedule works significantly better than the longer schedule used in STRG.The paper uses 24k iterations rather than STRG’s 50k-iteration schedule.
- LFB models consistently outperform models without LFB, including prior state-of-the-art, on both Charades validation and test sets.
- Charades gains are smaller than on the other datasets, partly because Charades uses a coarser video-level prediction task.
7. Discussion
The discussion shows that useful temporal context depends on the dataset: AVA benefits from minutes, EPIC-Kitchens from tens of seconds, and Charades from more than 10 seconds.
- AVA benefits from very long context lasting 2+ minutes.
- EPIC-Kitchens benefits from context spanning 15 to 60 seconds, while Charades benefits from extending support to 10+ seconds.
- The paper concludes that LFB supplies long-term supportive information and yields state-of-the-art results on AVA, EPIC-Kitchens, and Charades.
- Temporal-support comparisons account for each dataset’s best-performing feature-bank operator and use standard errors from five runs.
Appendix A. Backbone Architecture
The paper uses a non-local I3D backbone and examines how long-term modeling relates to short-term backbone improvements, including regularization for AVA.
- The backbone is a ResNet-50 I3D with non-local blocks, using limited temporal downsampling.
- The architecture specification assumes 32×224×224 input and places additional non-local blocks at specified residual stages.
- The person detector uses Faster R-CNN with a ResNeXt-101-FPN backbone and is fine-tuned on AVA bounding boxes.
- Long-term modeling through LFB and short-term backbone improvements improve different action classes, suggesting complementary effects.LFB helps interactive actions, whereas backbone improvements help more standalone actions.
- On AVA, dropout is insufficient to regularize STO, while randomly injected distractor features are effective.The paper reports no obvious overfitting for STO on the other datasets.
Appendix E. Training Schedule for EPIC-Kitchens
The noun models use a 50k-iteration training schedule with weight decay 10−6 and a learning rate of 0.001, reduced at iterations 40k and 45k.
- Noun models train for 50k iterations with weight decay 10−6.
- The noun-model learning rate starts at 0.001.
- The learning rate decreases by 10 times at iterations 40k and 45k.
Appendix F. EPIC-Kitchens Inference
EPIC-Kitchens inference combines independently trained verb and noun predictions using a training-frequency prior, while the appendix documents detector performance and complementary short- versus long-term modeling effects on AVA.
- Inference: Action probability is computed as the product of verb and noun softmax scores, weighted by a prior μ.
- Inference: The prior μ is estimated as the count of a verb–noun pair divided by the count of the noun in training annotations.
- Impact of LFB: LFB improves AVA mAP from 22.1 to 25.5, with larger gains on interactive actions such as singing, playing instruments, and computer work.
- Impact of backbone: Backbone improvement raises AVA mAP from 22.1 to 23.0 and produces larger gains on standalone actions such as crouching, reading, and handshaking.
- Complementarity: LFB and stronger backbones improve different action classes, suggesting complementary long-term and short-term modeling.
- Object detection: The detector achieves 2.4 AP on the new validation split, where many classes are unseen and many validation classes have zero instances.
Appendix H. Charades Training Schedule
Charades training uses a two-stage LFB procedure, and both NL′ variants consistently outperform STO and the baseline 3D CNN, with post-activation selected as default.
- Training schedule: Baseline Charades 3D CNNs use a 24k training schedule with learning rate 0.02, reduced at iteration 20k.
- Training schedule: LFB training first omits FBO, then adds FBO, freezes the backbone, and trains for an additional 12k iterations.
- NL′ variants: The Charades comparison evaluates pre-activation and post-activation NL′ variants.
- NL′ variants: LFB consistently outperforms STO and the baseline 3D CNN for both NL′ variants.
- NL′ variants: Post-activation is chosen as the Charades default because it performs more strongly, while AVA and EPIC-Kitchens show no noticeable difference between variants.