Source-linked AI summary
Multi-modal Dense Video Captioning
Vladimir Iashin, Esa Rahtu
TL;DR
Dense video captioning must localize events and describe them, yet most prior methods ignore audio and speech. MDVC uses Transformer-based fusion of visual, audio, and ASR-derived speech inputs, achieving competitive ActivityNet Captions results and substantial multimodal contributions in ablations.
Problem
Most dense video captioning methods rely solely on visual information despite audio and speech providing vital cues for understanding video environments.
Method
MDVC uses Transformer models to combine video frames, raw audio, and temporally aligned speech captions extracted by an ASR system.
Results
MDVC achieves competitive results to the current state-of-the-art on ActivityNet Captions, with ablations indicating substantial contributions from audio and speech.
Takeaways & Limitations
Audio and speech provide complementary information to video frames for dense video captioning, and MDVC supports any number of modalities.
Takeaways & Limitations
Incomplete dataset access makes strictly fair comparison difficult, so evaluation includes missing-feature inputs and a no-missings multimodal setup.
Abstract
from arXiv · showhide
Dense video captioning is a task of localizing interesting events from an untrimmed video and producing textual description (captions) for each localized event. Most of the previous works in dense video captioning are solely based on visual information and completely ignore the audio track. However, audio, and speech, in particular, are vital cues for a human observer in understanding an environment. In this paper, we present a new dense video captioning approach that is able to utilize any number of modalities for event description. Specifically, we show how audio and speech modalities may improve a dense video captioning model. We apply automatic speech recognition (ASR) system to obtain a temporally aligned textual description of the speech (similar to subtitles) and treat it as a separate input alongside video frames and the corresponding audio track. We formulate the captioning task as a machine translation problem and utilize recently proposed Transformer architecture to convert multi-modal input data into textual descriptions. We demonstrate the performance of our model on ActivityNet Captions dataset. The ablation studies indicate a considerable contribution from audio and speech components suggesting that these modalities contain substantial complementary information to video frames. Furthermore, we provide an in-depth analysis of the ActivityNet Caption results by leveraging the category tags obtained from original YouTube videos. Code is publicly available: github.com/v-iashin/MDVC
1. Introduction
Dense video captioning localizes events in untrimmed videos and describes each event, but prior systems largely rely only on visual information. MDVC adds audio and speech through ASR and evaluates their contribution on ActivityNet Captions.
- Dense video captioning first temporally localizes events in unconstrained footage, then generates a natural-language description for each event.
- Most recent dense video captioning systems cast captioning as machine translation and use Transformer architectures to model sequence dependencies.
- Prior work predominantly uses visual information, although audio and speech can provide vital cues for understanding video content.
- MDVC combines video frames, raw audio, and speech content, using ASR to extract time-aligned speech captions alongside the other inputs.
- On ActivityNet Captions, MDVC achieves competitive results to the current state-of-the-art, while ablations indicate substantial contributions from audio and speech.
2. Related Work
Dense video captioning evolved from template-based and recurrent encoder-decoder systems toward machine-translation formulations and Transformers. MDVC extends this line by scaling multimodal inputs to domain-free dense video captioning.
- Early video captioning used detected objects and predefined templates, while later methods formulated captioning as machine translation with neural encoder-decoder models.
- Subsequent work improved captioning through shared visual-textual memory, attention, reinforcement learning, semantic tags, additional modalities, and paragraph generation.
- ActivityNet Captions introduced dense video captioning as a task combining temporal event localization with textual descriptions for localized events.
- Transformer-based methods addressed recurrent models’ difficulty with long-term dependencies, while other approaches modeled object interactions and reduced caption redundancy.
- The closest prior Transformer model uses only visual features, whereas MDVC incorporates information from multiple modalities.
- Earlier multimodal efforts included visual-audio or video-speech systems, often limited by weak supervision or instructional-video domains.
- MDVC emphasizes speech and audio on a domain-free dataset and supports scaling to any number of modalities.
3. Proposed Framework
MDVC uses a two-stage pipeline that first localizes temporal events and then generates captions from visual, audio, and speech modalities. Individual Transformer encoders process modality features, while a decoder and generator produce caption words for each proposal.
- Two-stage framework: MDVC first obtains temporal event proposals, then generates a caption for each proposal.Bi-SST predicts event start and end locations before passing proposals to caption generation.
- Temporal Event Localization Module: Bi-SST uses 3D convolutional video features and bidirectional LSTM passes to estimate event-boundary confidence scores from past and future context.Forward and backward scores are fused and thresholded to form the final proposal set.
- Multi-modal Dense Video Captioning: Caption generation combines I3D visual features, VGGish audio features, and ASR-derived speech text as inputs to individual Transformer models.The ASR representation provides temporally aligned speech information alongside video and audio inputs.
- Feature Transformer: Each modality’s Transformer encoder maps proposal-aligned features to internal representations, while a decoder attends to encoder outputs and previously generated caption words.The decoder uses position-encoded caption history together with the encoder representation.
- Feature Transformer: A generator converts decoder representations into a vocabulary distribution and greedily selects successive words until an ending token is sampled.Caption generation starts with a special starting token, and both boundary tokens are included in the vocabulary.
- Feature Transformer: The Transformer architecture uses multi-headed attention to integrate information across representation subspaces and model dependencies beyond sequential past context.Encoder self-attention can access all states from the previous layer, unlike an RNN’s accumulation from past positions alone.
4. Experiments
Experiments evaluate MDVC on ActivityNet Captions using standard captioning metrics, architecture and modality ablations, baseline comparisons, and category-specific analysis. Results show strong performance under incomplete data and consistent benefits from combining visual, audio, and speech inputs.
- Dataset: ActivityNet Captions contains approximately 20k YouTube videos, averaging 3.65 localized captions, 13.65 words per caption, and two minutes per video.The dataset is split 50/25/25% for training, validation, and testing; validation videos have two annotations each.
- Dataset: 91% of training and validation videos were available, while only approximately 28% contained at least one speech segment.Missing videos and sparse speech availability constrain direct multimodal coverage.
- Comparison with Baseline Methods: MDVC achieves comparable or better performance despite less training data and 9% missing validation videos, and outperforms all baselines when all three modalities are present.The latter result holds for both ground-truth and learned event proposals.
- Ablation Studies: The Feature Transformer uses fewer parameters and achieves better dense video captioning performance than the Bi-GRU in visual-only experiments.Both models outperform the random-feature baseline.
- Ablation Studies: The three-modality model performs best among tested configurations, while concatenation with two fully connected layers outperforms probability averaging.Adding two fully connected layers to a visual-only model performs worse than any bimodal setup, indicating the gain is not explained solely by extra capacity.
- Ablation Studies: Audio-aware models capture speech details in qualitative captions that visual-only models miss, while category gains are consistent except for Film & Animation and Travel & Events.The exceptions may reflect mismatches between visual and audio tracks, such as music accompanying resort promotions or cartoon footage.
5. Conclusion
The paper introduces MDVC, a Transformer-based dense video captioning module that combines visual, audio, and speech modalities. Experiments on ActivityNet Captions show superior performance over visual-only models, while ablations verify the importance of audio and speech.
- MDVC encodes each modality for an event proposal and generates captions from combined visual, audio, and speech information.The model is based on the Transformer architecture.
- Experiments on ActivityNet Captions show superior performance over visual-only models in existing literature.
- Extensive ablation studies verify the contribution of audio and speech modalities to dense video captioning.
- The authors conclude that future video captioning systems should utilize multi-modal input.
6. Supplementary Material
The supplementary material presents qualitative multimodal captioning results, feature-extraction and implementation details, and comparisons with reinforcement-learning-based methods. It shows complementary strengths of audio, visual, and speech inputs, while documenting residual caption inaccuracies.
- Qualitative results: The supplement compares MDVC using audio-only, visual-only, and speech-plus-audio-plus-visual inputs on an ActivityNet Captions validation video.The example is a two-minute children's snowboarding advertisement with 12 human annotations and narrated speech.
- Qualitative results: Audio-only captions identify a speaking woman but miss snowboarding and children, whereas visual-only captions capture the scene while missing the speaker's gender.The video lacks snowboarding-related sound effects, making the audio-only setting challenging.
- Qualitative results: The multimodal model combines speech, audio, and visual cues to improve details such as identifying the speaker as a woman.Figure 6 illustrates the model producing “woman” rather than “man” when the lady speaks.
- Feature extraction: Audio features come from VGGish on 0.96-second log-mel spectrogram segments, while visual features come from I3D processing 24 RGB and optical-flow frames.The visual pipeline uses PWC-Net for optical flow and produces 1024-dimensional features per frame stack.
- Feature extraction: Speech input consists of temporally allocated English text segments, with every segment overlapping a proposal selected to provide coverage of speech during that proposal.Speech segments are represented similarly to captions, using word-order numbers.
- Comparative evaluation: Table 4 compares dense video captioning methods on ActivityNet Captions validation proposals using METEOR, including methods that directly optimize the metric with reinforcement learning.The supplementary discussion reports that reinforcement-learning methods generally perform better in METEOR and suggests MDVC could also benefit from them.