A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer

Vladimir Iashin; Esa Rahtu

Source-linked AI summary

A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer

Vladimir Iashin, Esa Rahtu

arXiv:2005.08271v2May 17, 2020cs.CVcs.CLcs.LGcs.SDeess.AS

TL;DR

Dense video captioning must localize events and describe them, but existing methods largely neglect audio and prior multimodal evidence is limited or domain-specific. The paper introduces a bi-modal Transformer with a shared bi-modal encoder and proposal generator, demonstrating state-of-the-art performance on ActivityNet Captions and gains over unimodal configurations.

Problem
Dense video captioning methods mainly exploit visual features, while prior multimodal results are limited or domain-specific.
Method
A bi-modal Transformer processes audio and visual sequences, while its pre-trained bi-modal encoder supports a multi-headed proposal generator and captioning module.
Results
The model achieves state-of-the-art results on F1 and BLEU metrics, with audio-visual configurations outperforming unimodal configurations across settings.
Takeaways & Limitations
The approach provides an integrated way to fuse audio and visual features for dense video captioning and to use the same encoder for proposals and captions.
Takeaways & Limitations
Prior speech-based evidence remains difficult to generalize because cooking videos are an ill-suited domain when subtitles can closely proxy captions.

Abstract

from arXiv · show

Dense video captioning aims to localize and describe important events in untrimmed videos. Existing methods mainly tackle this task by exploiting only visual features, while completely neglecting the audio track. Only a few prior works have utilized both modalities, yet they show poor results or demonstrate the importance on a dataset with a specific domain. In this paper, we introduce Bi-modal Transformer which generalizes the Transformer architecture for a bi-modal input. We show the effectiveness of the proposed model with audio and visual modalities on the dense video captioning task, yet the module is capable of digesting any two modalities in a sequence-to-sequence task. We also show that the pre-trained bi-modal encoder as a part of the bi-modal transformer can be used as a feature extractor for a simple proposal generation module. The performance is demonstrated on a challenging ActivityNet Captions dataset where our model achieves outstanding performance. The code is available: v-iashin.github.io/bmt

1 Introduction

Dense video captioning localizes events in untrimmed videos and describes each with a sentence, addressing the sparsity of single-caption approaches. This paper introduces a bi-modal Transformer that combines audio and visual cues for the task and reports state-of-the-art results on ActivityNet Captions.

1 Introduction: Dense video captioning first localizes events in untrimmed videos, then generates one natural-language sentence for each event.This addresses the sparsity of describing a long video with only one sentence.
1 Introduction: Existing dense video captioning models mainly use visual inputs despite the natural co-occurrence of audio and visual tracks.The paper identifies this underuse of audio as the motivation for its bi-modal approach.
1 Introduction: The proposed bi-modal Transformer combines audio and visual information for dense video captioning using attention inspired by the Transformer architecture.The proposal generator is inspired by the multi-headed design of YOLO.
1 Introduction: The model achieves state-of-the-art performance on BLEU@3–4 and F1 metrics on the open-domain ActivityNet Captions dataset.The result is reported for both the bi-modal dense video captioning module and the bi-modal proposal generator.

2 Related Work

Prior dense video captioning work developed event proposals, sequence-to-sequence captioning, reinforcement learning, weak supervision, and multimodal extensions. The paper positions its bi-modal Transformer as a more integrated approach that uses audio and visual information for both proposal generation and captioning.

2 Related Work: Earlier dense video captioning systems combined event proposal methods with recurrent or Transformer-based captioning architectures.Representative approaches used LSTM encoders, context-aware proposal networks, or Transformer outputs connected to proposal generation.
2 Related Work: Other research improved dense captioning through reinforcement-learning objectives or weak supervision, but weakly supervised results remained below supervised methods.Reinforcement learning targeted non-differentiable caption metrics, while weak supervision used non-localized captions with cycle consistency.
2.2 Multi-modal Dense Video Captioning: Multimodal approaches incorporated audio, speech, or both, but prior evidence was limited by specific domains or straightforward fusion designs.Cooking-video subtitles can act as accurate proxies for captions, while another approach fused modality-specific features by concatenation and retained a video-only proposal generator.
2.2 Multi-modal Dense Video Captioning: The proposed method uses a single bi-modal encoder for both proposal generation and captioning, and reports better results while using only visual and audio cues.Its proposal generator also uses both modalities and is reported to outperform the state of the art.

3 Our Framework

The framework combines a bi-modal Transformer for audio-visual captioning with a multi-headed proposal generator that localizes events before caption generation.

Bi-modal Encoder: The bi-modal encoder processes audio and visual feature sequences with self-attention, cross-modal attention, and position-wise fully connected layers.Its outputs are visual-attended audio and audio-attended visual features.
Bi-modal Decoder: The bi-modal decoder attends to encoder representations and previously generated caption words to predict the next word.It adds bi-modal encoder-decoder attention and a bridge layer to the standard decoder structure.
Event Proposal Generation Module: The proposal generator applies modality-specific proposal-generation heads at every timestamp and across anchor segment lengths.Heads use fully convolutional networks with modality-specific kernel sizes and preserve sequence length through padding and identity stride.
Event Proposal Generation Module: Proposal boundaries are computed from center offsets, anchor-scaled lengths, and objectness scores, then converted from feature-grid units into seconds.The final confidence is the sigmoid-transformed objectness score.
Event Proposal Generation Module: The common prediction pool is sorted by confidence, and the top-100 proposals are selected to clip input features for captioning.This connects event localization with the captioning module through proposal-conditioned feature sequences.

4 Experiments

On ActivityNet Captions, the bi-modal model achieves state-of-the-art captioning and proposal-generation performance, while ablations show benefits from audio-visual inputs and captioning-encoder pre-training for proposals.

4.1 Comparison to the State-of-the-art: In learned-proposal captioning, the model outperforms non-reward-maximization methods and leads BLEU metrics, while remaining competitive on METEOR.For ground-truth proposals, it is on par with the strongest METEOR result but trails that method on BLEU.
4.1 Comparison to the State-of-the-art: The model achieves state-of-the-art proposal-generation performance on ActivityNet Captions, with especially strong ground-truth segment coverage and prediction accuracy.The comparison uses precision, recall, and F1 on the validation set.
4.2 Ablation Study: The final training procedure pre-trains captioning, freezes its encoder, and then trains the proposal generator with that encoder.This procedure is termed “Cap →Prop” and is compared with separate training and the reverse order.
4.2 Ablation Study: Pre-training benefits proposal generation in the “Cap →Prop” setting, whereas captioning does not benefit from a pre-trained proposal-generator encoder.Captioning can perform worse when it uses that pre-trained encoder.
4.2 Ablation Study: Using both audio and visual modalities gives the best result in nearly all evaluated settings, although audio-only input is weakest.The visual-only to bi-modal gap remains consistent across settings, indicating that audio contributes useful cues.

5 Conclusion

The paper presents a bi-modal Transformer and proposal generator that fuse audio and visual features for dense video captioning. On ActivityNet Captions, the approach achieves state-of-the-art F1 and BLEU results and outperforms uni-modal configurations.

5 Conclusion: The proposed bi-modal Transformer and multi-headed proposal generator fuse audio and visual features for dense video captioning.The same bi-modal encoder supports both proposal generation and captioning.
5 Conclusion: On ActivityNet Captions, the model achieves state-of-the-art results on F1 and BLEU metrics.The ablation study reports that the bi-modal model outperforms uni-modal configurations in all settings.

6.1 Multi-headed Attention

Multi-headed attention uses multiple attention heads to learn distinct representation subspaces while preserving computational efficiency. Mapping modality inputs into a shared internal space enables attention between features with different original dimensions.

6.1 Multi-headed Attention: Attention uses query, key, and value sequences, with row-wise Softmax and scaling to keep gradients in a sufficient range.
6.1 Multi-headed Attention: Mapping inputs into an internal space allows attention between features whose original dimensions differ.The shared internal dimension is used when Dq ≠ Dk.
6.1 Multi-headed Attention: Multi-headed attention concatenates H attention heads and maps their outputs back to the query subspace.The heads use distinct trainable projections, while the output projection returns the concatenated representation to dimension Dq.

6.2 Feature Extraction

The model extracts pre-calculated audio, visual, and caption-token features with modality-specific pretrained networks and embeddings. These representations provide 128-dimensional audio, 1024-dimensional visual, and 300-dimensional decoder features.

6.2 Feature Extraction: Audio is represented as sequences of 128-dimensional VGGish features extracted from 0.96-second segments.VGGish was pretrained on AudioSet and processes log mel-scaled spectrograms.
6.2 Feature Extraction: Visual input is represented as sequences of 1024-dimensional I3D features, with each feature spanning 2.56 seconds of video.I3D uses stacks of 64 RGB and optical-flow frames and was pretrained on Kinetics.
6.2 Feature Extraction: Caption tokens are mapped to 300-dimensional GloVe embeddings, using the average vocabulary vector for missing tokens.The embeddings come from a Common Crawl vocabulary of 2.2 million words.
6.2 Feature Extraction: The bi-modal encoder uses audio and visual dimensions of 128 and 1024, while decoder inputs and outputs use dimension 300.

6.3 Implementation Details

Implementation uses padded, masked batches and a unified training procedure for captioning and proposal generation. The model uses a two-layer, four-head bi-modal Transformer and selects checkpoints by validation performance.

6.3 Implementation Details: Captioning and proposal-generation batches use sizes 32 and 16, with padding masked out during training.Proposal features are padded to 300 visual and 800 audio positions.
6.3 Implementation Details: The encoder and decoder use N = 2 layers and H = 4 attention heads, with an internal attention dimension of 1024.The implementation also uses label smoothing γ = 0.7 and dropout p = 0.1.
6.3 Implementation Details: Both model stages follow a unified training procedure rather than using different optimizers, learning rates, or scheduling techniques.The authors note that alternative combinations may improve performance.
6.3 Implementation Details: The captioning model peaked at epoch 26 and the proposal generator at epoch 17, with training times of 10 and 3.5 hours on an RTX 2080Ti.Models were selected using the best validation metric before proposals were captioned.

6.4 More Ablation Studies

Ablations and modality analyses evaluate filtered validation data, proposal conditions, and video categories. Bi-modal inputs consistently outperform uni-modal inputs, while replacing bi-modal attention causes a substantial metric decline.

6.4 More Ablation Studies: After excluding unavailable videos, other methods change by less than 2%, while the authors expect their method to gain about 9% from the filtering.The compared methods were trained on the full dataset, whereas this model used 91% of the training data.
6.4 More Ablation Studies: Applying the final captioning model to another method’s generated proposals isolates captioning performance from proposal generation.The comparison reports better captioning performance for this model on the filtered validation set.
6.4 More Ablation Studies: The category analysis uses YouTube labels and compares audio-only, visual-only, and bi-modal systems on the ActivityNet Captions validation subset.Categories are ordered by bi-modal performance with learned proposals, and their video counts are shown.
6.4 More Ablation Studies: Bi-modal models consistently outperform audio-only and visual-only models across all video categories and both ground-truth and learned-proposal settings.The analysis also indicates that visual input provides more cues than audio in nearly all cases.
6.4 More Ablation Studies: Replacing encoder bi-modal attention with self-attention causes a substantial decline across all reported metrics.The self-attention variant nevertheless remains stronger than the visual-only and audio-only models.

6.5 Qualitative Analysis

The bi-modal model captures speech-related content that the visual-only model misses, while producing generally meaningful captions alongside some visual-detail errors. Annotator captions remain more detailed than model-generated descriptions.

Qualitative comparison: The bi-modal and audio-only models captured the video's talking concept, which the video-only model neglected.The visual-only model's caption was still judged sensible overall.
Qualitative comparison: The bi-modal model's caption was not entirely erroneous despite differing from ground truth about whether the person talks.The video shows a verbose explanation of a martial arts movement, providing context for the prediction.
Qualitative comparison: The model incorrectly guessed the man's shirt color, possibly because a black punching bag appeared on screen.
Caption detail: Annotator captions were significantly more detailed, whereas model captions were more general.The paper identifies this as an issue requiring further attention for dense video captioning systems.