Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning

1. Introduction

• The paper motivates multimodal perception by noting that audio can clarify relationships and event stories that are less apparent from vision alone.It uses the example of a silent film to illustrate how missing sound can make a story's events harder to perceive.Source: §1 · p. 1
• Earlier audio-video work often addressed roughly 20-second videos with a single salient event, and audio's contribution was typically considered marginal.The authors identify high-level perceptual tasks as an area where learning from multiple modalities has been limited.Source: §1 · p. 1
• Dense event captioning concerns longer, more complex videos, usually two minutes or more, and the authors make sound a first-class perceptual modality for this task.They describe their work as the first to tackle dense event captioning with sound, to their knowledge.Source: §1 · p. 2
• The approach builds on weakly supervised dense event captioning, which avoids requiring temporal-segment annotations during training.Such annotations require caption labels and corresponding event start and end times, which the paper says are difficult and time-consuming to collect.Source: §1 · p. 2
• On ActivityNet Captions, combining audio and video improves state-of-the-art performance, while audio alone can nearly rival a visual model.The paper also compares audio representations and fusion strategies.Source: §1 · p. 2

Audio Feature Representations; Multi-modal Features in Video Analysis

• Audio representations and audio-visual methods have been explored for sound localization, audio event detection, and video analysis tasks.Related work includes sound-source localization [27] [31] [35], audio event detection [29] [36], and multimodal activity recognition [2].Source: §2 · p. 2

Multi-modal Features in Video Analysis

• Multimodal video captioning work has used attention, hypothesis-level integration, feature fusion, hierarchical networks, and modality selection.The cited approaches include [16], its extension, and methods in [15] [44] [38].Source: §2 · p. 2

Dense Event Captioning in Videos

• Dense event captioning was introduced in [20], and prior approaches include two-stage proposal-and-caption pipelines and joint localization-generation methods.Examples include proposal generation followed by captioning [26], a descriptiveness regression component [22], and JEDDi-Net [46].Source: §2 · pp. 2–3
• The weakly supervised framework in [12] removes ground-truth temporal segments and pairs event captioning with sentence localization in an iterative training approach.The paper builds on that framework and notes [12] was then the only weakly supervised dense event captioning method.Source: §2 · p. 3

3. Multi-modal Dense Event Captioning

• Weak supervision removes the need for ground-truth temporal event segments during training, and the method iteratively couples sentence localization with caption generation.Starting from random proposal segments, the generator produces captions, the localizer refines segments, and the generator then refines captions; the authors use one iteration following [12].Source: §3 · p. 3
• The sentence localizer encodes audio, video, and captions with GRU-based RNNs, applies crossing attention, then fuses attention features to produce temporal segments.The architecture's localizer takes the three modalities as inputs and generates a segment for each caption.Source: §3 · p. 3
• The caption generator clips audio and video features using the temporal segments, fuses their context, and decodes one caption per multimodal context.Its decoder is a GRU-based RNN, and the paper compares three context-fusion strategies.Source: §3 · p. 3

3.1. Feature Representation

• Audio and video feature selection is treated as a task-specific design choice, with the paper comparing representations for both modalities.The authors identify audio feature selection in particular as challenging.Source: §3.1 · p. 3

3.1.1 Audio Feature Processing

• The authors collected about 15,840 YouTube audio tracks for ActivityNet Captions and considered MFCC, CQT, and SoundNet representations.The dataset does not provide audio tracks; some original YouTube videos were no longer available.Source: §3.1.1 · p. 3
• MFCC and CQT features are converted to 512-dimensional representations with VGG-16, while SoundNet supplies 1000-dimensional features from its conv8 layer.MFCC uses a short-term power-spectrum representation; CQT uses geometrically spaced frequency bins with equal Q-factors.Source: §3.1.1 · p. 3

3.1.2 Video Feature Processing

• Video frames are encoded with the ActivityNet Captions dataset's pretrained C3D network into 500-dimensional features at a 16-frame time resolution.The resulting feature sequence has length Tv/δ, where δ is the frame interval for each feature.Source: §3.1.2 · p. 4

3.2. Weakly Supervised Model

• Weak supervision omits ground-truth temporal alignments, with the model assuming a one-to-one correspondence between captions and event segments.A sentence localizer maps captions to segments, while a caption generator maps segments to captions.Source: §3.2 · p. 4
• The model encodes visual, audio, and caption streams as sequences of recurrent features, with a context segment represented by its temporal center and length.The video/audio sequence has one feature vector per frame, while caption features correspond to words.Source: §3.2 · p. 4

3.2.1 Sentence Localizer

• The sentence localizer regresses temporal segments from caption and video or audio context features, using crossing attention and attention-feature fusion.It starts from 15 predefined temporal segments and predicts offsets to fine-tune their boundaries.Source: §3.2.1 · pp. 4–5

3.2.2 Caption Generator

• The caption generator softly clips audio and video features to each segment, fuses the clipped contexts, and decodes one caption per fused context.The continuous mask makes clipping differentiable; normalized weighted pooling approximates mean-pooling over clipped frames.Source: §3.2.2 · p. 5

3.3. Context Fusion

• The three context-merging strategies fuse video and audio features after normalized soft-mask clipping.Source: §3.3 · p. 5

Multiplicative Mixture Fusion

• Multiplicative mixture fusion adds the video and audio contexts, then concatenates their sum with both original contexts.The method is described as allowing the model to emphasize a more reliable modality and reduce emphasis on a less reliable one [25].Source: §3.3 · p. 5

Multi-modal Context Fusion

• Multimodal context fusion applies the fusion technique to segment-level audio and video features rather than full-video context.The strategy is described as similar to Eq. 6.Source: §3.3 · p. 5

MUTAN Fusion

• MUTAN fusion adapts a scheme proposed for visual question answering [6] to combine video and audio features, reducing their dimensions before modeling interactions.The dimension reduction uses the idea of Tucker decomposition.Source: §3.3 · pp. 5–6

3.4. Training Loss

• The training procedure uses cycle consistency for the sentence localizer and caption generator, and formulates temporal segment regression as classification [12].Source: §3.4 · p. 6
• The final loss combines caption reconstruction cross-entropy, segment reconstruction L2 loss, and temporal-regression cross-entropy, weighted by tunable parameters λs and λr.Caption reconstruction measures sentence similarity, while segment reconstruction measures similarity between temporal segments.Source: §3.4 · p. 6

4.1. Dataset

• The experiments use 8,026 training and 3,880 validation videos with available audio, out of 10,009 and 4,917 videos, respectively.ActivityNet Captions averages 3.65 captions per video, but audio tracks had to be collected from YouTube because the dataset provides C3D features rather than original videos.Source: §4.1 · p. 6

4.2. Experiment Setup and Implementation Details

• Training first pretrains the caption generator, then jointly trains it with the sentence localizer; testing filters 15 random temporal proposals before captioning.The pretrained model is initialized from the caption generator, and final-model testing applies one round of fixed-point iteration and IoU filtering.Source: §4.2 · p. 6

4.3. Experiment Results

• MFCC and SoundNet audio representations produce comparable audio-only results, and MUTAN is reported as the most appropriate of the tested fusion strategies.The audio-only experiments include pretrained and final models; the fusion comparison uses MFCC and C3D features.Source: Table 2 · p. 7, §4.3 · p. 8
• 9% and 13% improvement on Bleu@3 and Bleu@4, respectively, is reported for the multimodal approach over the state-of-the-art unimodal method [12].MUTAN is selected as the most appropriate fusion strategy; the comparison uses MFCC or SoundNet audio with C3D video features.Source: Table 3 · p. 7, §4.3 · p. 8
• In a human forced-choice evaluation, the multimodal model receives a 60.67% preference rate; qualitative examples also favor it over unimodal models in caption quality and segment accuracy.The human evaluation compares the multimodal model with the visual-only model using 15 validation videos and 20 participants.Source: Figure 4 · p. 7, §4.3 · p. 8
• The final model sometimes misses event beginnings and usually generates around two event captions, so it does not detect all events in the weakly supervised setting.Source: §4.3 · p. 8

• Combining audio and visual features achieves performance better than the state-of-the-art unimodal visual model.Source: §5 · p. 8
• The proposed approach generates dense event captions from video using both audio and visual features in a weakly supervised setting.Source: §5 · p. 8
• The experiments compare different feature representations and context-fusion strategies, and indicate that audio features play a vital role.Source: §5 · p. 8