Audio-Visual Event Localization in Unconstrained Videos

1 Introduction

• Prior in-the-wild audio-visual representation work assumed matched audio and visual content, leaving joint representations for unconstrained-video understanding unexplored.The authors describe the collective research questions as not previously studied systematically as a whole.Source: §1 · pp. 1–2
• The paper defines three tasks: supervised and weakly supervised event localization, plus event-agnostic cross-modality localization between synchronized audio and visual segments.Weak supervision uses video-level event tags without temporal boundaries; cross-modality localization finds corresponding content in the other modality.Source: §1 · p. 2
• The proposed methods include audio-guided visual attention, a dual multimodal residual fusion network, and audio-visual distance learning for cross-modality localization.The paper also introduces a dataset of more than 4,000 annotated unconstrained videos.Source: §1 · pp. 2–3
• Experiments find that joint audio-visual modeling outperforms independent modeling, attention captures sounding-object regions, temporal alignment matters for fusion, and cross-modal correlations support localization.The study also reports tractable localization under noisy conditions and that attention can distinguish audio-visual unrelated videos.Source: §1 · p. 3

Sound event detection

• Sound event detection methods and datasets focus on audio signals, so sound-only datasets are unsuitable for audio-visual scene understanding.The related-work discussion cites HMM, GMM, DNN, and BLSTM approaches and TUT Acoustic Scenes.Source: §2 · pp. 3–4

Temporal action localization

• Unlike temporal action localization, this work includes motionless sounding objects and models both audio and visual domains, while adapting ED-TCN as a comparison.Action-localization work commonly treats sliding windows as action candidates; other cited methods include deep action proposals and S-CNN.Source: §2 · p. 4

Multimodal machine learning

• Prior multimodal learning studies joint representations and fusion, whereas this work investigates fusion of learned audio-visual features for localization.Related approaches include statistical models, MKL, graphical models, and deep networks.Source: §2 · p. 4

Vision-and-sound modeling

• Earlier vision-and-sound work explored representation learning, spatial sound-source localization, synthesis, and image-scene/audio-caption associations; this paper systematically studies event localization.Examples include visual-teacher sound representations, ambient-sound supervision, audio-visual correspondence, and sound-source localization [38].Source: §2 · p. 4

3.1 AVE: The Audio-Visual Event Dataset

• AVE is a subset of AudioSet with 4,143 videos across 28 event categories, temporally labeled with audio-visual event boundaries.Each video contains at least one two-second audio-visual event; 66.4% of videos contain events spanning the full ten seconds.Source: §3.1 · pp. 4–5

3.2 Fully and Weakly-Supervised Event Localization

• Fully supervised localization predicts event labels segment by segment from audio and visual tracks, using one-second non-overlapping segments.The label set includes the AVE event categories plus a background label.Source: §3.2 · p. 5
• Weakly supervised localization trains from video-level event tags without temporal boundaries but still predicts segment-level labels at test time.The task is intended to reduce reliance on well-annotated data for audio, visual, and audio-visual modeling.Source: §3.2 · p. 5

3.3 Cross-Modality Localization

• Event-agnostic cross-modality localization finds synchronized visual content from an audio segment or synchronized audio from a video segment.Evaluation uses short-event videos whose audio-visual event lengths are all under ten seconds.Source: §3.3 · pp. 5–6

4.1 Audio-Visual Event Localization Network

• The network extracts CNN features, applies audio-guided visual attention, models each modality with a separate LSTM, fuses hidden states, and predicts segment labels with a shared Softmax layer.Source: §4.1 · pp. 6–7

4.2 Audio-Guided Visual Attention

• Audio-guided attention learns from visual feature maps and audio vectors to weight visual regions; its maps capture sound-source locations and improve temporal localization accuracy.Source: §4.2 · pp. 7–8

4.3 Audio-Visual Feature Fusion

• DMRN updates both modalities to preserve original and add cross-modal information; one block suffices, while late fusion outperforms early fusion.Source: §4.3 · p. 8

4.4 Weakly-Supervised Event Localization

• With only video-level labels, MIL pooling aggregates segment-pair predictions into video-level predictions, while testing assigns event categories to individual segments.Source: §4.4 · p. 9

• The audio-visual distance learning network measures distance for a visual-audio pair and uses sliding windows for visual localization from audio.Source: §5 · p. 9
• The matching window starts where cumulative audio-to-visual segment distances are minimized; the matched visual content can also be used for audio localization.Source: §5 · p. 9
• Contrastive loss pushes dissimilar pairs apart when their distance is below margin th; pairs beyond the margin do not contribute to the loss.Source: §5 · pp. 9–10

6.1 Visual and Audio Representations

• Visual features use VGG-19 pool5 maps averaged across 16 frames, while audio uses 128-D features; combining C3D features yields no significant improvement.C3D is explored for spatiotemporal visual features.Source: §6.1 · p. 10

6.2 Baselines and Evaluation Metrics

• ED-TCN reaches 46.9% overall accuracy, versus 55.3% for the V model; comparisons also include single-modality baselines and ED-TCN.Overall accuracy is used for supervised and weakly supervised localization; cross-modality accuracy counts exact matches.Source: §6.2 · pp. 10, 12, §6.3 · p. 12

6.3 Experimental Comparisons

• Audio-visual models outperform their single-modality counterparts in both supervised and weakly supervised localization.Audio alone generally outperforms visual alone, though visual features are stronger for some noisy outdoor events.Source: §6.3 · pp. 12–13, Table 1 · p. 12
• 3.3% absolute improvement for V-att over V and 1.3% for A+V-att over A+V accompany attention maps that focus on sounding regions.The maps can capture localized sound sources and spread across background regions when no audio-visual event is present.Source: Figure 5 · p. 11, Figure 6 · p. 11, Table 1 · p. 12
• Late-fusion DMRN outperforms all compared fusion methods, and DMRFE further improves performance.The compared methods use the same audio and visual inputs.Source: §6.3 · p. 13, Table 2 · p. 13
• Late fusion outperforms early and decision fusion, consistent with temporal modeling before fusion helping learn alignments between modalities.Late fusion also outperforms decision fusion, which combines lower-dimensional features after separate processing.Source: §6.3 · p. 14
• Weakly supervised localization is promising despite supervised models performing better, supporting MIL-based localization from noisy supervision.The weakly supervised setting is evaluated alongside fully supervised localization.Source: §6.3 · p. 14
• AVDL outperforms DCCA by a large margin on both A2V and V2A under an exact-match evaluation.The results support cross-modality localization on unconstrained videos.Source: §6.3 · p. 14

• Joint auditory-visual modeling outperforms independent modeling, and localization remains tractable under noisy conditions.Source: §7 · p. 14
• Audio-guided visual attention captures sound-source semantics and can distinguish audio-visual unrelated videos.Source: §7 · p. 14
• Temporal alignment matters for audio-visual fusion, the dual residual network is capable of fusion, and strong cross-modal correlations enable cross-modality localization.Table 3 reports accuracy for visual localization from an audio-segment query (A2V) and audio localization from a visual-segment query (V2A).Source: §7 · p. 14, Table 3 · p. 14

• The supplementary material describes dataset gathering, algorithm implementation, and additional experiments.Source: arXiv:1803.08842v1 [cs.CV] 23 Mar 2018 · p. 19

1 AVE: The Audio-Visual Event Dataset

• The AVE dataset contains 4,143 videos across 28 categories, with manually annotated audio-visual event boundaries.Source: §1 · p. 19

1.1 Gathering and Preparing Dataset

• AudioSet includes many clips unsuitable for localization because their audio and visual content may be unrelated or sound may be absent.Source: §1.1 · p. 19
• Annotators selected videos from about 10,000 clips in 34 AudioSet categories, marking boundaries to one-second resolution when the visible source was audible for at least two seconds.Source: §1.1 · p. 19

• The 4,143 videos are split into 3,339 training, 402 validation, and 402 test videos; cross-modality evaluation samples only test videos with events shorter than 10 seconds.Models use PyTorch and Keras with TensorFlow, Adam optimization, an LSTM hidden size of 128, and a contrastive-loss margin of 2.0.Source: §2 · p. 20

3.1 Spatio-Temporal Feature for Audio-Visual Event Localization

• C3D-related models perform well mainly for categories with rich action and motion, while adding C3D to AVs yields no noticeable category-wide improvement.The authors therefore use spatial visual features alone in the main paper.Source: Table 1 · p. 20, §3.1 · p. 22
• On AVE, A outperforms Vs, both substantially outperform Vc3d, and AVs+c3d is only slightly better than AVs.The reported overall accuracy comparison favors audio and spatial visual features over C3D features.Source: Table 2 · p. 21, §3.1 · p. 22
• C3D features come from the pool5 layer of a Sport1M-pretrained network and are globally average-pooled into spatio-temporal feature vectors.The experiment tests whether video temporal information helps localization.Source: §3.1 · p. 22

3.2 Different Attention Mechanisms

• Attention mechanisms act as weighted global pooling; co-attention combines attended audio and visual feature vectors.The experiments compare visual-guided audio attention, audio-guided visual attention, and their co-attention combination.Source: §3.2 · p. 22
• V-att outperforms V, but A′-att does not beat A′ and A′+V-co-att is slightly worse than A′+V.A′ uses pooling-layer audio features, unlike A's 128-D last fully connected-layer features; the A′ model is worse than A in the reported comparison.Source: Table 3 · p. 21, §3.2 · p. 22
• Audio-guided visual attention locates semantic regions containing sounding objects, whereas visual-guided audio attention captures frequency patterns that are difficult to interpret.The visual-guided audio attention uses log-mel spectrogram patches representing one-second waveforms.Source: §3.2 · pp. 22–23