Source-linked AI summary
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, Dima Damen
TL;DR
Egocentric action recognition must combine complementary audio and visual cues that may occur at different times, while prior fusion strategies often aggregate or synchronize modalities prematurely. The paper introduces an end-to-end Temporal Binding Network that fuses RGB, Flow, and Audio within temporal windows before aggregation, achieving state-of-the-art EPIC-Kitchens leaderboard results and demonstrating audio’s value.
Problem
Audio and visual cues in egocentric actions can be asynchronous, while existing methods commonly aggregate modalities before fusion or fuse synchronized inputs.
Method
The Temporal Binding Network samples RGB, Flow, and Audio within temporal binding windows, performs mid-level fusion before temporal aggregation, and shares weights across segments.
Results
The method achieves state-of-the-art results on the EPIC-Kitchens public leaderboard for both seen and unseen test sets, outperforming individual modalities and late fusion.
Takeaways & Limitations
Audio is competitive with appearance in egocentric recognition, and audio-visual fusion improves recognition of actions and accompanying objects.
Takeaways & Limitations
Future work includes learning to adjust temporal binding windows over time and using class-specific windows.
Abstract
from arXiv · showhide
We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i.e. the combination of modalities within a range of temporal offsets. We train the architecture with three modalities -- RGB, Flow and Audio -- and combine them with mid-level fusion alongside sparse temporal sampling of fused representations. In contrast with previous works, modalities are fused before temporal aggregation, with shared modality and fusion weights over time. Our proposed architecture is trained end-to-end, outperforming individual modalities as well as late-fusion of modalities. We demonstrate the importance of audio in egocentric vision, on per-class basis, for identifying actions as well as interacting objects. Our method achieves state of the art results on both the seen and unseen test sets of the largest egocentric dataset: EPIC-Kitchens, on all metrics using the public leaderboard.
1. Introduction
Egocentric recordings provide informative but temporally asynchronous audio-visual cues. The paper proposes binding modalities within a temporal window to exploit these complementary signals for action recognition.
- Motivation: Audio can identify some actions and interacting objects, disambiguate visually similar actions, and capture audible events outside the camera’s view.Sound is also absent for some actions and may include irrelevant background audio.
- Motivation: Existing multimodal methods typically aggregate each modality before fusion or fuse synchronized inputs, limiting their handling of asynchronous signals.The egg-cracking example illustrates that sound, motion, and appearance can be informative at different temporal positions.
- Proposed idea: A temporal binding window allows modalities to be fused across a range of temporal offsets rather than at one synchronized instant.The proposed window is motivated by evidence that humans perceptually bind sensory inputs within task-dependent temporal ranges.
- Contributions: The proposed end-to-end Temporal Binding Network uses mid-level fusion and reports state-of-the-art results on seen and unseen EPIC-Kitchens test sets.The paper also reports audio efficacy, an advantage over late fusion, and robustness to irrelevant sounds.
2. Related Work
Prior work commonly fuses visual or egocentric modalities through late fusion, while audio-visual action recognition has largely been studied outside the egocentric setting. This paper reports that audio is substantially more useful for egocentric recognition than prior third-person results suggest.
- Visual Fusion for AR: Two-stream action-recognition systems commonly combine appearance and motion through late fusion after independent processing.Related approaches also use temporal models or predefined offsets for cross-modal interactions.
- Fusion in Egocentric AR: Egocentric action-recognition methods have extended late fusion with streams for hands, objects, head motion, masks, and saliency.The cited approaches generally rely on small-scale egocentric datasets.
- Audio-Visual Learning: Audio-visual learning research has addressed representation learning, sound-source localization, source separation, and visual-question answering.These works provide broader audio-visual context but are not primarily egocentric action-recognition systems.
- Audio-Visual Learning: Third-person benchmarks reported audio as the least informative modality, including 16% audio accuracy versus 80% spatial and 78% motion accuracy on UCF101.Similar conclusions were reported for AVA and Kinetics.
- Audio-Visual Learning: On EPIC-Kitchens, audio performs comparably to appearance, and audio-visual fusion improves recognition of both actions and accompanying objects.This contrasts with the weaker audio results reported for third-person datasets.
3. The Temporal Binding Network
The Temporal Binding Network samples modalities within temporal windows, fuses their representations before temporal aggregation, and shares processing weights across sparse temporal segments. This design supports asynchronous multimodal inputs while contrasting with late and synchronous fusion.
- Fusion alternatives: Late-fusion architectures aggregate each modality temporally before combining their predictions.In the formal setup, unimodal feature extractors and temporal aggregation precede multimodal fusion.
- Fusion alternatives: Synchronous fusion combines modalities at matching or approximately matching time steps before temporal aggregation.Varying frame rates and representation sizes make exact synchronization difficult.
- Temporal binding: TBN fuses modalities within a Temporal Binding Window, constraining all temporal offsets to a finite range rather than requiring synchronization.With three modalities, each input can have a different offset while remaining within the same ±b window.
- Temporal binding: Sampling within a temporal window supports multiple frame rates and varied temporal shifts, unlike methods using predefined temporal differences.The approach is intended to scale to many modalities and accommodate different action speeds.
- Sparse temporal sampling: TBN divides each video into K equal segments, samples one modality sparsely per segment, and randomly selects the other modalities within its centered TBW.Multiple TBWs may overlap, and their predictions are averaged.
- Mid-level fusion: Per-modality ConvNets extract mid-level features that are concatenated and mapped through a fully connected layer to produce multimodal predictions per TBW.All modalities are trained jointly through backpropagation to the ConvNet inputs.
- Weight sharing: TBN shares convolutional, fusion, and class-prediction weights across temporal segments.This preserves a common processing scheme while applying the model across sparse temporal samples.
- Relation to TSN: Unlike TSN, TBN learns the modality combination jointly and produces one fused prediction per TBW rather than independently aggregated modality predictions.The comparison concerns both where fusion occurs and how the modalities are trained.
4. Experiments
Experiments on EPIC-Kitchens evaluate TBN across modalities, fusion strategies, temporal-binding widths, and seen/unseen splits. Fusion improves over individual modalities and late fusion, while audio provides complementary information and TBN achieves strong leaderboard results.
- Dataset and setup: EPIC-Kitchens contains 39,596 action segments from 32 participants, with 125 verb and 331 noun classes evaluated on seen and unseen kitchen splits.Actions combine a verb and noun; S1 shares environments between training and testing, whereas S2 holds out complete sequences from four participants.
- Per-class modality analysis: Noun classes are often predicted well from RGB alone, whereas verbs rely more on Flow and Audio because they reflect temporal progression and distinctive sounds.Audio is particularly useful for some verbs and for nouns associated with distinctive sounds or materials that make sounds when manipulated.
- Single-modal and multimodal performance: Audio is comparable to RGB on top-1 verb accuracy, while Flow is substantially stronger than RGB on the unseen split.On the seen split, RGB and Flow perform comparatively; overall, fusion outperforms every single modality.
- Per-class modality analysis: Fusion gains are larger for verbs than nouns and vary by class, with complementary modalities especially benefiting tail classes.For example, the verb ‘spray’ reaches fusion 0.54 versus RGB 0.09, Flow 0, and Audio 0.3; fusion also significantly boosts tail classes.
- Audio efficacy and robustness: Adding audio to RGB+Flow increases top-5 action accuracy by 5% on S1 and 4% on S2, with audio also improving diagonal confidence and reducing confusion among major verb classes.Segments with irrelevant sounds remain challenging, but the authors report that irrelevant audio is not the source of confusion and that the network is robust to noisy audio.
- Fusion and temporal binding: Concatenation generally outperforms more complex mid-level fusion strategies, while TBN performance remains robust across larger temporal-binding windows.The reported synchronization baseline is only approximate because RGB, Flow, and Audio have different sampling rates.
- Comparison with the state of the art: A single TBN improves S1 top-1 verb, noun, and action accuracy by 9%, 10%, and 11%, and S2 by 6%, 5%, and 6%, respectively, over the baseline.An ensemble of five TBNs adds up to 3% on top-1 metrics; at submission, the ensemble improved top-1 verb results by 11% on both splits and ranked second in the 2019 challenge.
5. Conclusion
TBN flexibly combines RGB, Flow, and Audio, improving performance over individual modalities and exceeding TSN on the same data. The authors identify adaptive and class-specific temporal binding windows as future directions.
- TBN combines RGB, Flow, and Audio to improve performance across individual modalities.
- Future work includes learning to adjust temporal binding windows over time and implementing class-specific windows.
Appendices
The appendices provide supplementary qualitative examples, leaderboard results, and per-class analyses comparing performance with and without audio.
- Supplementary material describes qualitative examples presented in the video appendix.
- The appendices include leaderboard results and per-class analyses with and without audio.
A. Qualitative results
Qualitative evaluation uses 14 held-out untrimmed training videos and compares predictions from RGB, Flow, Audio, and the TBN single model against ground truth.
- 14 held-out untrimmed training videos provide qualitative examples from publicly available training footage.
- Each example compares ground truth with predictions from RGB, Flow, Audio, and the TBN Single Model.
B. Action Recognition Challenge - Public Leaderboard Results
On the EPIC-Kitchens public leaderboard, the TBN single model outperformed other submissions on both test sets, while an ensemble improved results further using varied TBW widths. The final TBN ensemble ranked second.
- The TBN Single Model outperformed all other submissions by a clear margin on test sets S1 and S2.
- An ensemble of TBNs trained with different TBW widths further improved the leaderboard results.
- The TBN Ensemble ranked second when the challenge concluded.
C. Per-class Multi-modal Fusion Results
Per-class results show that multimodal fusion improves recognition unevenly, with especially large gains for underrepresented tail classes and clearer benefits for some audio-informative actions and objects.
- Adding audio increases diagonal confidence and decreases off-diagonal confusion in the largest-15 verb and noun confusion matrices.
- 63% improvement on tail verbs versus 34% on top-10% verbs, showing stronger fusion gains for underrepresented verb classes.
- 50% improvement on tail nouns versus 15% on top-10% nouns, indicating larger fusion benefits for underrepresented noun classes.
- Audio outperforms RGB and Flow on tail verbs, while nouns benefit most from audio when they have distinct sounds such as ‘switch’ and ‘paper’.
- Fusion improves results when individual modalities are either all strong, as for ‘open’ and ‘fridge’, or all weak, as for ‘scoop’.