Source-linked AI summary
Weakly Supervised Dense Event Captioning in Videos
Xuguang Duan, Wenbing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu, Junzhou Huang
TL;DR
Dense event captioning requires locating and describing all events, but existing methods rely on costly temporal annotations. This paper introduces WS-DEC, which uses one-to-one caption–segment correspondence and a cycle between sentence localization and caption generation. Experiments support its effectiveness for dense event captioning and sentence localization, while its event coverage is limited when videos contain many small events.
Problem
Dense event captioning needs temporal segment annotations and caption correspondence for training, creating a costly supervision requirement.
Method
WS-DEC uses one-to-one caption–segment correspondence and cyclic sentence localization followed by caption reconstruction without ground-truth segments.
Results
Experiments verify the method's effectiveness for dense event captioning and sentence localization.
Takeaways & Limitations
The approach removes accurate temporal-coordinate annotations and also provides an unsupervised method for sentence localization.
Takeaways & Limitations
The model usually generates only 2 to 3 descriptions, limiting its ability to capture videos containing many small events.
Abstract
from arXiv · showhide
Dense event captioning aims to detect and describe all events of interest contained in a video. Despite the advanced development in this area, existing methods tackle this task by making use of dense temporal annotations, which is dramatically source-consuming. This paper formulates a new problem: weakly supervised dense event captioning, which does not require temporal segment annotations for model training. Our solution is based on the one-to-one correspondence assumption, each caption describes one temporal segment, and each temporal segment has one caption, which holds in current benchmark datasets and most real-world cases. We decompose the problem into a pair of dual problems: event captioning and sentence localization and present a cycle system to train our model. Extensive experimental results are provided to demonstrate the ability of our model on both dense event captioning and sentence localization in videos.
1 Introduction
Dense Event Captioning seeks to locate and describe every event in a video, but existing training requires costly temporal segment annotations. This paper introduces WS-DEC, using only captions and a cycle between sentence localization and caption generation.
- Motivation: Dense Event Captioning locates all events in a video and generates a caption for each event.Unlike tasks using a few tags or labels, it describes video content in natural language.
- Motivation: Existing DEC training requires captions, temporal start and end times, and correspondence between each event and its caption, making annotation source-consuming.Weak supervision omits event coordinates and caption correspondence during training.
- Problem formulation: WS-DEC trains dense event captioning using caption annotations without temporal segment annotations.The paper defines weak supervision as incomplete supervision rather than less information.
- Method: The method assumes one-to-one correspondence between captions and temporal segments, then cycles from sentence localization to caption reconstruction.The reconstruction objective repeatedly optimizes the caption generator and sentence localizer without ground-truth segments.
- Evaluation: The paper evaluates its approach on ActivityNet Captions for dense event captioning and sentence localization.The reported experiments verify dense event captioning ability and sentence localization accuracy.
2 Related Work
Prior work developed video captioning, dense event captioning, and sentence localization as related but largely separate directions. Dense event captioning typically detects events before describing them, while sentence localization uses scan-and-localize or temporal-coordinate regression frameworks.
- Video captioning: Video captioning evolved from mean-pooled frame features toward recurrent networks with attention for longer videos.Mean pooling works for short clips but degrades as video length increases.
- Dense Event Captioning: Dense event captioning methods generally follow a detection-and-description framework.Prior approaches combine event proposal or detection mechanisms with caption generators such as context-aware S2VT or LSTM-A.
- Sentence localization: Sentence localization research expanded from constrained visual domains to real-world videos with deep-learning models.The reviewed approaches include candidate-clip matching, coarsely sampled clip regression, and direct temporal-coordinate regression.
3 The Proposed Method
The method jointly learns sentence localization and event captioning under a one-to-one segment–caption assumption, using cycle reconstruction to train without segment labels. At test time, random candidate segments are refined through the dual system and captioned densely.
- 3.1 Formulation: The framework formulates sentence localization and event captioning as dual mappings between videos, captions, and temporal segments.Localization maps (V, C_i) to S_i, while captioning maps (V, S_i) to C_i.
- 3.1 Formulation: The method assumes each caption corresponds to one temporal segment and each segment has one caption.This one-to-one correspondence makes the two dual problems applicable simultaneously.
- 3.1 Formulation: Training localizes a segment from a caption, regenerates the caption from that segment, and optimizes reconstruction loss.The nested functions form an auto-encoder in which the segment is omitted from the final caption reconstruction.
- 3.1 Formulation: At test time, random candidate segments are iteratively refined toward fixed-point solutions and then passed to the caption generator.The paper reports using one iteration in experiments and generating captions from the refined segments.
- 3.1 Formulation: The combined objective adds a segment-consistency loss to the caption reconstruction loss with trade-off parameter λ_s.The segment loss compares the original and regenerated localizations, while λ_s controls its contribution.
- 3.2 Network Design: The network uses a sentence localizer with crossing attention and a caption generator with differentiable soft temporal clipping.Crossing attention computes video–caption interactions, while soft clipping avoids nondifferentiable frame selection during end-to-end training.
4 Experiments
Experiments evaluate the weakly supervised model for dense event captioning and sentence localization, using ActivityNet Captions and comparisons with supervised, unsupervised, and model variants. Results show improved event detection over random proposals and baselines, while qualitative examples reveal remaining difficulties with event boundaries and videos containing many short events.
- Experimental setup: Experiments use ActivityNet Captions, a benchmark containing 20,000 videos and approximately 100,000 annotated events.Each video contains 3.65 annotated events on average.
- Experimental setup: The evaluation reports METEOR, CIDEr, Rouge-L, and Bleu@N across tIoU thresholds of 0.3, 0.5, 0.7, and 0.9.Scores are computed only when predicted and ground-truth segments overlap beyond the relevant threshold.
- Experimental setup: The study compares the proposed method with two fully supervised dense-captioning methods and internal variants that remove or alter localization components.Variants include pretrained random-segment captioning, global temporal regression, and classification without regression.
- Dense event captioning: Random testing segments produce small variance across seeds, while increasing their number slightly improves performance; the final setting uses N_r = 15.The authors select N_r = 15 as a trade-off between complexity and performance.
- Dense event captioning: The model detects events more accurately than the random proposal model and is generally better than the two baseline methods.Figure 2 reports recall of detected events across testing segments and tIoU thresholds.
- Dense event captioning: Qualitative results show more accurate and detailed descriptions than the pretrained model, but the system may miss event beginnings and fail to capture all events in videos with many short events.The authors associate missed beginnings with reliance on a temporal segment’s final hidden state.
- Sentence localization: For sentence localization, the supervised implementation reaches performance similar to ABLR, while the unsupervised model substantially outperforms CTRL.The result indicates that the model can learn meaningful temporal segments from indirect losses.
5 Conclusion and Future Work
The paper introduces WS-DEC and an efficient cycle-based solution, reporting effectiveness on dense event captioning and sentence localization. It also identifies web-video evaluation and broader weakly supervised applications as future directions.
- WS-DEC eliminates accurate temporal-coordinate annotations and supports using videos in the wild to reduce annotation cost.
- The proposed cycle process and fixed-point iteration effectively address WS-DEC and provide unsupervised sentence localization.
- Future work includes evaluating the model directly on web videos and applying the method to other weakly supervised tasks.