Source-linked AI summary
MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition
Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer
TL;DR
Video recognition systems handle snapshots and short clips but have limited support for longer temporal context. MeMViT processes videos sequentially while caching memory from prior clips, achieving longer-term modeling with efficient scaling and improved benchmark performance.
Problem
Existing video recognition systems accurately parse snapshots or short clips but remain limited in connecting visual information across longer time ranges.
Method
MeMViT processes consecutive short clips online and caches prior representations as memory for long-term context.
Results
30× longer input duration requires only 4.5% more compute, and MeMViT achieves large gains and state-of-the-art results across AVA and EPIC-Kitchens-100 tasks.
Takeaways & Limitations
MeMViT provides a more efficient way to extend video models to long-term visual modeling than simply adding more input frames.
Takeaways & Limitations
Caching full key and value tensors can retain redundant information, while memory compression increases memory during training because full memory tensors are jointly used.
Abstract
from arXiv · showhide
While today's video recognition systems parse snapshots or short clips accurately, they cannot connect the dots and reason across a longer range of time yet. Most existing video architectures can only process <5 seconds of a video without hitting the computation or memory bottlenecks. In this paper, we propose a new strategy to overcome this challenge. Instead of trying to process more frames at once like most existing methods, we propose to process videos in an online fashion and cache "memory" at each iteration. Through the memory, the model can reference prior context for long-term modeling, with only a marginal cost. Based on this idea, we build MeMViT, a Memory-augmented Multiscale Vision Transformer, that has a temporal support 30x longer than existing models with only 4.5% more compute; traditional methods need >3,000% more compute to do the same. On a wide range of settings, the increased temporal support enabled by MeMViT brings large gains in recognition accuracy consistently. MeMViT obtains state-of-the-art results on the AVA, EPIC-Kitchens-100 action classification, and action anticipation datasets. Code and models are available at https://github.com/facebookresearch/memvit.
1. Introduction
MeMViT addresses the limited temporal range of video recognition by processing videos online and caching prior representations as memory. This enables substantially longer temporal support with far less additional compute than increasing the input frame count.
- Current video systems accurately parse snapshots or short periods such as 5 seconds but remain limited for longer-term visual understanding.
- MeMViT processes videos online while maintaining reusable memory that provides prior context at each time step.The model processes consecutive clips and caches representations for future recognition.
- 30× longer input duration requires only 4.5% more compute with MeMViT, whereas increasing the number of frames requires >3,000% more compute.
- MeMViT uses transformer keys and values from the current clip and past memory, with multilayer attention producing a longer receptive field.
- A jointly trained memory compression module reduces the memory footprint by retaining cues considered important for future recognition.
- MeMViT achieves large gains over short-term counterparts and state-of-the-art results on AVA and EPIC-Kitchens-100 classification and anticipation tasks.
2. Related Work
Long-term video models seek to capture patterns over durations exceeding 30 seconds while avoiding prohibitive computation. MeMViT contributes an end-to-end memory design that supports arbitrary-layer features with one standalone backbone.
- Long-term video models target patterns in videos over durations such as >30 seconds.
- Existing approaches reduce cost by modeling pre-computed features, designing efficient architectures, or using memory-like end-to-end designs.
- Long-term feature-bank methods reference supportive context features but use only final-layer features and require two backbones plus two rounds of training and inference.
- MeMViT models features at arbitrary layers with minimal changes to standard training methods and requires one standalone backbone.
- Attention-based online video modeling can reference arbitrary past times without the forgetfulness associated with recurrent networks.
- Video memory caching is challenging because video tensors are much larger than those processed by long-range language models.
3. Preliminaries
The paper builds MeMViT on the Multiscale Vision Transformer and frames long-video processing as sequential short-clip processing with cached memory. The preliminaries describe ViT attention and MViT’s multiscale and pooling mechanisms.
- MeMViT is based on the MViT architecture, while the proposed techniques are applicable to other ViT-based video architectures.
- MeMViT processes a long video as consecutive short T × H × W clips and caches representations from each processed clip.
- ViT embeds an image into N non-overlapping patches represented by a tensor X0 ∈ R^N×d, then models patch interactions through transformer layers.
- Transformer attention linearly projects an input tensor X into queries Q, keys K, and values V before producing an output tensor Z.
- MViT learns multiscale representations by moving from fine-grained small patches to later high-level larger patches through strided pooling.
- MViT uses pooling attention, and the paper modifies it by swapping the order of the linear layer and pooling operation.
- Increasing input frames is the conventional way to extend video duration, but it substantially increases computational cost.
4. MeMViT for Efficient Long-Term Modeling
MeMViT processes consecutive short clips online while caching prior representations as memory, allowing current clips to attend to earlier context. Learned compression reduces memory overhead, while the design extends receptive fields hierarchically across layers and supports practical long-term processing.
- Online Processing: MeMViT processes consecutive short clips sequentially and caches representations from earlier iterations for long-term context.Videos are read online during training and inference; memory from a previous video is masked at video boundaries.
- Memory Attention and Caching: Transformer keys and values from the current and previous M iterations are concatenated so queries attend to both present and past information.A stop-gradient operator prevents further backpropagation dependency into the past.
- Memory Attention and Caching: The receptive field grows with both per-layer memory length M and network depth L because each layer attends further into the past.This hierarchical construction increases temporal coverage with depth.
- Limitations: The basic implementation caches full key and value tensors, which may contain redundant information and increase memory consumption and cost as temporal support grows.Compression is introduced to retain important information while reducing this overhead.
- Memory Compression: Learnable compression modules reduce cached key and value tensors by retaining information considered important for future recognition.The proposed pipelined design trains compression one step at a time, caches only the immediately previous memory uncompressed, and reuses compressed memory thereafter.
- Efficiency: The compressed design adds only a constant compression cost while drastically reducing caching and attention costs for other steps.The paper gives a default example of a 16× reduction and reports that training and inference overhead otherwise grows with M rather than traditional scaling behavior.
5. Experiments
Experiments show that MeMViT scales temporal support efficiently, maintains gains across design choices and settings, and outperforms prior methods on AVA and EPIC-Kitchens tasks.
- 5.1. Scaling Strategies: MeMViT scales to longer temporal support with lower training and inference memory, runtime, and FLOPs than increasing input frames.The comparison varies memory length M against baseline scaling that increases the input frame count T.
- 5.1. Scaling Strategies: Under equal computational costs, MeMViT achieves clearly better accuracy, while compression improves the accuracy-efficiency trade-off over uncompressed memory.These results support the use of memory with compression for practical long-term video modeling.
- 5.2. Ablation Experiments: Memory augmentation improves AVA performance by 1.7–2.3% absolute mAP, with M=2 producing a 36-second receptive field and the best AVA result.M=2 is used as the default for subsequent AVA experiments.
- 5.2. Ablation Experiments: A 4×2×2 temporal, height, and width downsampling configuration performs strongly, and learned compression improves accuracy over no compression.The results suggest that the compression module can retain useful cues while reducing memory.
- 5.2. Ablation Experiments: Augmenting 50% of layers performs best while saving computation, especially when memory-augmented layers are distributed uniformly through the network.Memory at every attention layer is unnecessary in this ablation.
- 5.3. Generalization Analysis: MeMViT generalizes across model sizes, pretraining datasets, and EPIC-Kitchens tasks, achieving large gains and state-of-the-art results on the evaluated benchmarks.On EPIC-Kitchens, MeMViT improves noun classification by 2.4%, verbs in anticipation by 3.5%, and tail verbs by 3.7%.
- 5.4. State-of-the-Art Comparison: MeMViT reaches 35.4 mAP on AVA after higher-resolution fine-tuning and outperforms prior work on EPIC-Kitchens classification and anticipation.Against AVT+ in anticipation, the reported margins are +1.8% action, +4.0% verb, and +5.0% noun.
6. Conclusion
MeMViT addresses long-term video understanding by extending existing video models with memory, improving efficiency and accuracy. The approach is presented as a general strategy for transformer-based video models.
- MeMViT scales more efficiently than extending existing models by adding more input frames, while achieving better accuracy.The paper presents memory-based modeling as a practical approach for long-term visual understanding.
A.1. Architecture Specifications
MeMViT is based on the Multiscale Vision Transformer and incorporates relative positional embeddings for spatiotemporal memory. Its architecture is specified for the default and larger model variants.
- MeMViT’s architecture is based on MViT with improvements from Li et al.; exact specifications are provided in Table A.1.The table covers the default MeMViT-16, 16×4 model and the MeMViT-24, 32×3 variant.
- Relative positional embeddings encode relationships between query tokens and key/value tokens at different spatiotemporal positions.The embeddings interact with queries according to the relative positions between the queries and keys/values.
- The relative positional embedding implementation omits additional embeddings on values because preliminary experiments found no accuracy improvement.
- Relative positional embeddings are decomposed across temporal, vertical, and horizontal dimensions to represent spatiotemporal token relationships.Rt, Rh, and Rw represent temporal, frame-height, and frame-width relative embeddings, respectively.
A.2. Kinetics Pre-training Details
MeMViT is pre-trained with a progressive video-length strategy that moves from short sampled clips to full Kinetics videos. This schedule is intended to support spatial learning early and longer-video adaptation later.
- MeMViT pre-training progressively increases video length from one randomly sampled clip to the full 10-second Kinetics video.The strategy exposes the model to diverse spatial patterns early and longer videos later.
A.3. AVA Experiments
The AVA experiments use a Faster R-CNN person detector and an augmented output head. These implementation choices are described as part of the experimental setup.
- The AVA experiments use a Faster R-CNN with a ResNeXt-101-FPN backbone as the person detector.The detector obtains 93.9 AP@50 on the AVA validation set.
- The AVA output head adds an MViT transformer layer before the linear classifier instead of using only a linear head.The added transformer layer operates without pooling because the tokens are already RoI-pooled, and it improves accuracy.
A.4. EPIC-Kitchens-100 Experiments
The EPIC-Kitchens-100 experiments specify training settings, causal modifications for anticipation, and separate prediction heads for classification and anticipation. Additional implementation-choice ablations are reported in Table A.2.
- Training: EPIC-Kitchens models are trained with AdamW for 30 epochs using a 0.0002 base learning rate, 0.05 weight decay, and batch size 128.Other hyperparameters follow the Kinetics recipe for MViT, and anticipation models are fine-tuned from classification models.
- Memory setting: When MeMViT processes a one-clip video, it effectively reduces to a short-term MViT because no previous-step memory is available.This defines the one-clip setting as a no-memory fallback condition.
- Causal anticipation: For anticipation, MeMViT is made causal so predictions depend only on current or past observed-video content.The modifications include causal pooling, masked attention, causal convolutions, and removal of the global classification token.
- Prediction heads: Action classification uses separate verb and noun heads, whereas anticipation predicts the action directly and marginalizes probabilities for verb and noun outputs.Both designs follow prior or standard practice described by the authors.
- Ablation: Table A.2 presents additional ablations of implementation choices on the default baseline model.The paper labels this analysis as a detailed model-detail ablation.