Source-linked AI summary
Dual-modality seq2seq network for audio-visual event localization
Yan-Bo Lin, Yu-Jhe Li, Yu-Chiang Frank Wang
TL;DR
Audio-visual event localization requires identifying events that are both visible and audible in a video. The paper proposes AVSDN, which jointly models audio and visual features over time using global and local information in a sequence-to-sequence framework. Experiments report favorable performance against recent approaches in both fully and weakly supervised settings.
Problem
Audio-visual event localization must match visible and audible components of the same event, while jointly handling learned audio and visual features remains challenging.
Method
AVSDN jointly takes audio and visual data at each time segment and uses sequence-to-sequence modeling of global and local event information.
Results
The proposed network performs favorably against state-of-the-art approaches on the AVE dataset and supports both fully and weakly supervised localization.
Takeaways & Limitations
AVSDN provides one framework for audio-visual event localization across fully and weakly supervised settings.
Abstract
from arXiv · showhide
Audio-visual event localization requires one to identify theevent which is both visible and audible in a video (eitherat a frame or video level). To address this task, we pro-pose a deep neural network named Audio-Visual sequence-to-sequence dual network (AVSDN). By jointly taking bothaudio and visual features at each time segment as inputs, ourproposed model learns global and local event information ina sequence to sequence manner, which can be realized in ei-ther fully supervised or weakly supervised settings. Empiricalresults confirm that our proposed method performs favorablyagainst recent deep learning approaches in both settings.
1. INTRODUCTION
The paper addresses audiovisual event localization, where an event must be both visible and audible, by proposing AVSDN to model audio and visual information jointly across time. The network supports fully and weakly supervised learning and reports favorable performance against state-of-the-art approaches on the AVE dataset.
- Motivation: Audio-visual event localization identifies events that are simultaneously visible and audible in video segments.The AVE dataset investigates supervised, weakly supervised, and cross-modality temporal localization tasks.
- Approach: AVSDN is an end-to-end sequence-to-sequence framework that jointly processes audio and visual data at each time segment.It uses an encoder, fusion network, and decoder to learn and use global and local event information.
- Training settings: AVSDN can be learned in both fully supervised and weakly supervised settings using frame- or video-level event labels.The paper describes this flexibility as a central contribution of the proposed trainable network.
- Approach: The model conditions event localization on global features formed from cross-modality information across time.The decoder uses the encoded global features together with local audio and visual information.
- Results: The proposed model performs favorably against state-of-the-art approaches on the challenging AVE dataset.This is reported as the paper's experimental headline result.
2. PROPOSED METHOD
AVSDN is an end-to-end seq2seq framework that processes audio and visual inputs over time, fuses global and local cross-modality information, and supports supervised and weakly supervised event localization.
- Supervised event localization: In supervised localization, training uses segment-wise labels for each second, with non-background events requiring jointly observed audio and visual events.The label space contains C event categories plus one background category.
- Weakly-supervised event localization: In weakly supervised localization, video-level labels are formed by averaging segment predictions, reducing reliance on densely annotated labels.The averaged prediction is optimized through binary cross-entropy for both supervision settings.
- Encoder: AVSDN uses separate audio and visual LSTMs to encode segment-level inputs into global representations from the final time step.Visual frames come from ResNet-152, while one-second audio segments are represented using VGGish features.
- Fusion network: The fusion network combines the audio and visual encoder states into a shared video-event representation used to initialize the decoder LSTM.The fused hidden and cell states provide the decoder's initial states.
- Decoder: The decoder concatenates global fused representations with local audio and visual segment features to generate segment-wise event labels.This design captures both spatial and temporal cross-modality representations.
3. EXPERIMENTS
Experiments evaluate AVSDN on the AVE dataset using frame-wise accuracy in fully and weakly supervised settings. The model outperforms the compared methods, while ablations examine multimodal decoder initialization and global event representations.
- Dataset: The AVE dataset contains 4,143 videos across 28 categories, with audio-visual events labeled every second.Examples include church bells, dog barking, trucks, buses, clocks, and violins.
- Evaluation: Frame-wise accuracy measures the percentage of correct event matchings across testing frames in both supervision settings.
- Comparison results: AVSDN performs better than the compared state-of-the-art methods in fully supervised and weakly supervised localization.The supervised comparison uses frame-level labels during training, whereas the weakly supervised comparison uses only video-level labels.
- Ablation studies: Multimodal decoder initialization improves results over initialization from only visual or audio content.The ablation also reports that additional losses for guiding the encoder’s final hidden states are unnecessary.
4. CONCLUSION
The paper concludes that AVSDN performs video event localization with local audio-visual features and integrated global representations in both fully and weakly supervised settings.
- Conclusion: AVSDN combines local audio-visual features with an integrated global representation for sequence-to-sequence event localization.
- Conclusion: The experimental results verify the proposed network and its design for fully and weakly supervised video event localization.