Source-linked AI summary
End-to-End Dense Video Captioning with Masked Transformer
Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, Caiming Xiong
TL;DR
Dense video captioning must detect and describe events in untrimmed videos, but prior systems train proposal and captioning modules separately or alternately. This paper proposes an end-to-end Transformer with differentiable proposal masking and self-attention. It reports significant improvement over RNN-based models on event proposal and captioning tasks, evaluated on ActivityNet Captions and YouCookII.
Problem
Prior dense-captioning systems separate event proposal and captioning training, preventing language descriptions from directly influencing event proposals.
Method
An end-to-end Transformer uses an encoder, anchor-based proposal decoder, and captioning decoder whose masking network converts proposals into differentiable masks.
Results
The model achieves significant performance improvement over RNN-based models on both event proposal and captioning tasks and is demonstrated on ActivityNet Captions and YouCookII.
Takeaways & Limitations
Differentiable masking keeps proposal and captioning consistent during training, while self-attention provides the paper’s non-recurrent approach to dense video captioning.
Takeaways & Limitations
The reported results are described as overly optimistic because the evaluation targets best-situation performance.
Abstract
from arXiv · showhide
Dense video captioning aims to generate text descriptions for all events in an untrimmed video. This involves both detecting and describing events. Therefore, all previous methods on dense video captioning tackle this problem by building two models, i.e. an event proposal and a captioning model, for these two sub-problems. The models are either trained separately or in alternation. This prevents direct influence of the language description to the event proposal, which is important for generating accurate descriptions. To address this problem, we propose an end-to-end transformer model for dense video captioning. The encoder encodes the video into appropriate representations. The proposal decoder decodes from the encoding with different anchors to form video event proposals. The captioning decoder employs a masking network to restrict its attention to the proposal event over the encoding feature. This masking network converts the event proposal to a differentiable mask, which ensures the consistency between the proposal and captioning during training. In addition, our model employs a self-attention mechanism, which enables the use of efficient non-recurrent structure during encoding and leads to performance improvements. We demonstrate the effectiveness of this end-to-end model on ActivityNet Captions and YouCookII datasets, where we achieved 10.12 and 6.58 METEOR score, respectively.
1. Introduction
Dense video captioning compresses untrimmed videos into event-level natural-language descriptions, but prior systems separate proposal generation from captioning. This paper introduces an end-to-end Transformer with differentiable masking and self-attention to connect these stages.
- Motivation: Dense video captioning localizes temporal events and describes them with natural-language sentences, preserving language content that video summarization loses.This supports faster understanding of video while retaining language components important in instructional content.
- Problem: Prior methods train event-proposal and captioning modules independently or alternately, preventing language information from directly influencing event proposals.The proposal module therefore cannot adjust localization based on the generated description.
- Approach: The proposed end-to-end model uses an encoder, anchor-based proposal decoder, and captioning decoder with a differentiable mask over each proposal.The mask restricts captioning attention to the proposal event and enables consistent joint training.
- Approach: Self-attention is used in the encoder and decoder to learn long-range dependencies without recurrent time unrolling.The paper states that shorter attention paths facilitate dependency learning and that Transformer trains and tests faster than recurrent models.
- Results: The model achieves competitive results on the ActivityNet Captions and YouCookII datasets.The supplied introduction identifies both datasets but does not provide their numerical scores.
2. Related Work
Related work covers neural video captioning and temporal action proposals. The paper differs from prior dense-captioning work by jointly learning event localization and description rather than combining separate modules without direct language influence.
- Image and Video Captioning: Recent video-captioning methods commonly encode frames with CNNs and generate language with recurrent decoders, varying mainly in frame encoding and attention mechanisms.The paper uses temporal attention and additionally applies self-attention within each module.
- Temporal Action Proposals: Temporal action proposals localize action-agnostic segments in long untrimmed videos and are commonly formulated as binary classification problems.Existing approaches differ in how proposals are generated and discriminated.
- Dense Video Captioning: Earlier dense-captioning work learns event locations and captions but combines proposal and captioning modules without the end-to-end consistency mechanism proposed here.The related-work passage identifies Krishna et al. as the most similar prior work before describing that combination.
3. Preliminary
The preliminary section presents Transformer as an encoder-decoder architecture built from multi-head attention and feed-forward layers. It distinguishes cross-module attention from self-attention and motivates the model’s non-recurrent structure.
- Attention: Scaled dot-product attention produces a weighted sum of values, with weights determined by query-key dot products.Keys and values are packed into matrices K and V in practice.
- Attention: Multi-head attention runs H parallel scaled dot-product attention layers, called heads, whose outputs are combined through learned projections.Each head performs an independent dot-product attention operation.
- Attention Types: Cross-module attention uses decoder hidden states as queries and encoder hidden states as keys and values, whereas self-attention takes all three from the same hidden layer.This distinction defines the attention configurations used in Transformer modules.
- Transformer: Figure 2 depicts a Transformer with a 1-layer encoder and a 1-layer decoder.The caption specifies the layer counts but not additional visual encodings.
- Transformer: Transformer combines multi-head attention with pointwise feed-forward layers, and its encoder and decoder stack multiple such building blocks.The feed-forward layer uses two linear projections with ReLU activation and can be viewed as two size-one convolution layers.
4. End-to-End Dense Video Captioning
The model jointly detects and describes video events through an end-to-end encoder–decoder architecture. Differentiable masking links proposal boundaries to captioning, while self-attention provides non-recurrent temporal context.
- Model Architecture: The model contains a video encoder, proposal decoder, and captioning decoder with a mask prediction network.The proposal decoder outputs event proposals, while the captioning decoder generates descriptions for selected proposals.
- Video Encoder: Self-attention lets each temporal representation incorporate information from all video time steps through short information paths.This makes gradient updates independent of temporal distance compared with recurrent models.
- Proposal Decoder: The proposal decoder uses explicit anchors, event scores, and center and length offsets to predict temporal event boundaries.Temporal convolutions generate proposal scores and offsets from the encoder output.
- Captioning Decoder: The captioning decoder applies a differentiable mask derived from proposal boundaries to restrict visual attention to the current event.The encoder propagates forward again during decoding so each layer’s representation is specific to the proposal.
- Captioning Decoder: A gated masking formulation combines the learned continuous mask with the proposal-derived discrete mask using the proposal confidence score.The continuous mask supplements the proposal mask when proposal confidence is low.
- Model Learning: The model is trained end-to-end with boundary regression, mask prediction, event classification, and captioning losses.The differentiable masking scheme allows captioning and proposal learning to remain consistent during training.
5. Implementation Details
The implementation uses temporal convolutions for proposal prediction and a Transformer configuration with high-dimensional representations and multi-head attention.
- Proposal Decoder: Proposal temporal-convolution kernels range from 1 to 251, with stride factor s set to 50.
- Transformer Configuration: The Transformer uses model dimension d = 1024, feed-forward hidden size 2048, and 8 attention heads.Dropout is applied in residual, attention, and visual-input embedding layers.
6. Experiments
Experiments compare the proposed Transformer models with recurrent and challenge baselines on ActivityNet Captions and YouCookII, including learned and ground-truth proposals. Results show stronger performance for Transformer-based models, while learned proposals are notably weaker on YouCookII and long-range modeling benefits longer events.
- Experimental setup: The evaluation covers ActivityNet Captions and YouCookII, using validation results and, for ActivityNet, challenge test-set comparisons.The experiments include learned event proposals, ground-truth proposals, event-proposal recall, and qualitative captioning results.
- ActivityNet results: All proposed models outperform LSTM-based models by a large margin on ActivityNet Captions.The authors attribute this result to improved modeling of long-range dependencies.
- YouCookII results: The Transformer model outperforms the LSTM baseline by a significant amount on YouCookII, but learned-proposal results are much worse than on ActivityNet.The authors suggest that small utensils and ingredients are difficult to detect with global visual features and propose incorporating object detectors or trackers.
- Qualitative results: The proposed method generates captions with more relevant semantic information than the baseline in qualitative ActivityNet results.Figure 3 uses color bars for different events and colored text to highlight event-relevant content.
- Event proposal analysis: A 4.5% improvement is reported over the recurrent-nets-based event-proposal model, while the Transformer is more accurate than Bi-LSTM when fewer proposals are allowed.With 100 proposals, Transformer and Bi-LSTM models yield similar recall; the recall curve is also examined at tIoU threshold 0.8.
- Captioning analysis: The proposed Masked Transformer exceeds the baseline by more than 1 METEOR point when captioning uses ground-truth event segments.This setting isolates captioning performance because the event proposals are accurate during inference.
7. Conclusion
The paper presents an end-to-end dense video captioning model with an encoder, proposal decoder, and captioning decoder. Differentiable masking aligns proposals with captioning, while self-attention supports the model’s non-recurrent design and improves performance on two datasets.
- The model combines an encoder with proposal and captioning decoders in a single end-to-end dense video captioning system.The proposal decoder forms event proposals, while the captioning decoder describes them.
- A differentiable masking network restricts captioning attention to each proposal event and maintains proposal-captioning consistency during training.
- Self-attention is used for dense video captioning and avoids an RNN-based model.
- The model achieves significant performance improvements on both event proposal and captioning tasks compared with RNN-based models.
- Effectiveness is demonstrated on the ActivityNet Captions and YouCookII datasets.
8. Appendix
The appendix reports implementation settings, self-attention ablations, short-event evaluation, and qualitative visualizations. These analyses examine optimization choices, model depth, long-range modeling, proposal confidence, and cross-module attention.
- Implementation: The end-to-end model uses SGD with Nesterov momentum, while proposal-only and captioning-only models use Adam with learning rate 0.0001.Transformer dropout is 0.2, visual-input dropout is 0.1, and the end-to-end loss coefficients are λ1, λ2, λ3, λ4 = 10, 1, 1, 0.25.
- Additional Results: Self-attention significantly affects performance in LSTM-baseline ablations, especially when applied in the language decoder.
- Additional Results: The captioning-only models vary little with layer count on ground-truth segments, so the experiments use a 2-layer transformer for lower cost and greater flexibility with approximate proposals.
- Additional Results: Short-event evaluation tests whether self-attention’s long-range modeling also preserves performance on activities lasting at most 15 seconds.
- Additional Qualitative Results: Mask visualizations compare high-confidence proposals above 0.9 with low-confidence proposals below 0.7.
- Additional Qualitative Results: Cross-module attention visualizations place generated words along the x axis and sampled frames along the y axis.Higher-level self-attention weights tend to be flatter than lower-level weights.
- Additional Qualitative Results: Qualitative YouCookII results include only the first six events in the second example and show challenges from small, ambiguous objects.