MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition

1. Introduction

• Existing action-recognition methods often use a single modality or treat modalities separately, partly because expressive cross-modal interactions are difficult to model.Advances in multimodal transformers motivate MM-ViT’s attempt to interpret visual and audio concepts together.Source: §1 · p. 1
• Modeling temporal structure efficiently remains difficult as action videos exhibit large variations and complexities.Earlier approaches aggregate features from 2D CNN frame processing, while 3D and pseudo-3D CNNs jointly convolve over space and time.Source: §1 · p. 1
• Compressed-video methods use motion vectors and residuals to avoid costly optical flow, but often treat modalities separately and omit audio.The paper notes these methods can run at least an order of magnitude faster than optical-flow competitors.Source: §1 · p. 2
• MM-ViT applies a pure transformer to compressed videos, combining I-frames, motion vectors, residuals, and audio while factorizing attention across space, time, and modality.It also introduces three cross-modal attention mechanisms for learning inter-modal interactions.Source: §1 · p. 2

2. Related Work

• CNN approaches aggregate frame-level features or jointly learn spatial-temporal features; multimodal variants add streams such as optical flow, audio, or skeleton estimates.3D convolution can improve accuracy but is computationally heavy, prompting pseudo-3D alternatives.Source: §2 · p. 2
• Compressed-video methods replace expensive optical flow with motion vectors, sometimes adding residuals or optical-flow supervision, but ignore rich inter-modal relations and audio.Other multimodal transformer work adds streams for joint vision-language, vision-audio, or vision-language-audio representations; MM-ViT instead uses one transformer for intra- and inter-modal features.Source: §2 · p. 2
• MM-ViT extends the ViT design to compressed video and audio, and outperforms Timesformer and ViViT on three action-recognition benchmarks while using fewer FLOPs.Timesformer and ViViT extend spatial attention to video’s spatial-temporal volume and factorize attention to manage computational cost.Source: §2 · p. 3

3.1. Representations in Compressed Videos

• Compressed video supplies I-frames, motion vectors, residuals, and audio; motion vectors approximate coarse optical flow, while residuals retain pixel differences after motion compensation.The paper describes these alongside P-frames, which contain motion vectors and residuals.Source: §3.1 · p. 3

3.2. Embedding Compressed Video Clips

• Visual patches receive learnable embeddings and spatiotemporal positional encodings, while VGGish projects audio segments into 128-dimensional vectors for alignment with the video.The audio waveform is partitioned into T segments before projection.Source: §3.2 · p. 3

3.3. Multi-modal Video Transformers

• MM-ViT compares joint attention with factorized variants: MM-ViT II reduces per-patch cost to O(N · |S| + T · |S|), and MM-ViT III to O(N + T + |S|).MM-ViT II uses an additional MSA operation; MM-ViT III factorizes attention over modality as well as space and time.Source: §3.3 · pp. 3–5
• Merged-Attention lets a modality query use all modalities’ keys and values, Co-Attention excludes its own modality, and Shift-Merge mixes value portions at matching spatial and temporal indices.Shift-Merge discards queries and keys; MM-ViT IV further restricts attention to local windows and adds convolution to strengthen neighboring-window connections.Source: §3.3 · p. 5

4.1. Setup

• Across UCF-101, SSv2, and Kinetics-600, videos were converted to MPEG-4; inference used eight sampled visual triplets and aligned audio except on SSv2.SSv2 has no audio stream; inference averages predictions from left, centre, and right spatial crops.Source: §4.1 · pp. 5–6

4.2. Ablation Studies

• Merged-Attention achieves the best accuracy on UCF-101 and SSv2; Shift-Merge Attention is comparably accurate and more efficient, while local-view MM-ViT IV drops 3.11% on UCF-101 and 4.72% on SSv2.The authors suggest more sophisticated cross-window connections may mitigate information loss from local attention views.Source: §4.2 · p. 6
• MM-ViT II improves on MM-ViT I by 0.83% on UCF-101 with 32% less FLOPs, while MM-ViT III (Merged-Att.) improves on MM-ViT II by 1.75% in accuracy with 3.5% less FLOPs.The comparison reports consistent accuracy and efficiency gains from factorizing self-attention over the input dimensions.Source: Table 1 · p. 6, §4.2 · p. 6
• Temporal⇒cross-modal⇒spatial is the best tested attention order on UCF-101, with temporal attention before spatial attention slightly but consistently outperforming the reverse.The authors attribute this pattern to temporal clues helping distinguish actions with similar appearance.Source: Table 2 · p. 6, §4.2 · p. 6
• On UCF-101, removing the I-frame lowers top-1 accuracy by 4.11%, while removing motion vectors, residuals, or audio causes drops up to 2.54%, 2.54%, and 2.42%, respectively.The authors suggest audio contributes dynamics and contextual temporal information useful for action recognition.Source: Table 3 · p. 6, §4.2 · p. 6

4.3. Comparison to State-of-the-Art

• On UCF-101, MM-ViT outperforms methods using the same compressed-video modalities by up to 6.9% in top-1 accuracy, and surpasses all CNN alternatives, with or without optical flow.Adding audio improves top-1 accuracy by 2.1%, and Kinetics-600 pre-training improves it by 3.5%.Source: §4.3 · p. 7
• On SSv2, MM-ViT exceeds Timesformer and ViViT with fewer inference FLOPs, beats single-RGB CNNs by ≥1.5% in top-1 accuracy, but trails two flow-based CNNs.The authors attribute the motion-feature benefit to the temporally heavy nature of SSv2; MM-ViT avoids optical-flow computation and storage.Source: Table 5 · p. 7, §4.3 · p. 7
• Video transformers still incur relatively large computational overhead compared with several efficient CNN architectures.The authors identify improving transformer computational and memory efficiency as future work.Source: §4.3 · p. 7
• 83.5% top-1 accuracy on Kinetics-600 (MM-ViT, T = 16), 1.3% above Timesformer and 0.5% above ViViT, with greater computational efficiency.It also exceeds CNN alternatives using either single RGB or additional flow information.Source: Table 6 · p. 7, §4.3 · p. 7

4.4. Qualitative Results

• Attention Rollout highlights action-relevant regions, including the eye and brush, and semantically aligned “lower eyelid” text for “Apply Eye Makeup.”The paper presents these visualizations as qualitative evidence of spatial-temporal-audio reasoning.Source: §4.4 · p. 8, Figure 3 · p. 8

• Across three public benchmarks, MM-ViT outperforms competing uni-modal video transformers in both efficiency and accuracy, and matches or exceeds CNNs assisted by computationally expensive optical flow.Source: §5 · p. 8
• MM-ViT performs action recognition in the compressed-video domain, factorizing self-attention over the input space to use rich multi-modal information at reasonable computational cost.Source: §5 · p. 8
• The experiments confirm the effectiveness of MM-ViT’s cross-modal attention mechanisms for learning inter-modal relations.Source: §5 · p. 8