Source-linked AI summary
Unifying Event Detection and Captioning as Sequence Generation via Pre-Training
Qi Zhang, Yuqing Song, Qin Jin
TL;DR
Dense video captioning divides the work of localizing events and describing them, while earlier methods may overlook temporal relations among events. This paper makes event detection a context-conditioned sequence-generation task and unifies detection and captioning with pre-training. On ActivityNet, it reports outperforming state-of-the-art methods on both tasks, with further gains from out-of-domain data.
Problem
Earlier dense video captioning methods often handled detection and captioning separately, and event detectors commonly ignored temporal relations between events.
Source: §1 · pp. 1–2Method
The model generates events one by one conditioned on previous events and unifies detection and captioning in a video-event-text pre-training and fine-tuning framework.
Source: §1 · pp. 2–3Results
On ActivityNet, the model significantly outperforms state-of-the-art methods on both event detection and captioning, with additional gains from out-of-domain pre-training data.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
The model generates more diverse events similar to human annotations while exploiting temporal relationships between events.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Dense video captioning aims to generate corresponding text descriptions for a series of events in the untrimmed video, which can be divided into two sub-tasks, event detection and event captioning. Unlike previous works that tackle the two sub-tasks separately, recent works have focused on enhancing the inter-task association between the two sub-tasks. However, designing inter-task interactions for event detection and captioning is not trivial due to the large differences in their task specific solutions. Besides, previous event detection methods normally ignore temporal dependencies between events, leading to event redundancy or inconsistency problems. To tackle above the two defects, in this paper, we define event detection as a sequence generation task and propose a unified pre-training and fine-tuning framework to naturally enhance the inter-task association between event detection and captioning. Since the model predicts each event with previous events as context, the inter-dependency between events is fully exploited and thus our model can detect more diverse and consistent events in the video. Experiments on the ActivityNet dataset show that our model outperforms the state-of-the-art methods, and can be further boosted when pre-trained on extra large-scale video-text data. Code is available at \url{https://github.com/QiQAng/UEDVC}.
1 Introduction
Dense video captioning couples event detection and captioning, but designing their interaction is difficult and earlier detectors overlook temporal relations between events. The paper reframes detection as context-conditioned event generation and unifies both tasks through pre-training, reporting gains on ActivityNet.
1 Introduction
- Earlier approaches separately handle event detection and captioning, while designing interactions between them remains difficult because their task-specific solutions differ.Prior work explored stronger inter-task association, but the differing task solutions make it difficult for the subtasks to fully benefit from each other.Source: §1 · pp. 1–2
- Conventional event detectors ignore temporal relationships among events, which can produce redundant candidates or events with excessive overlap.The paper notes that instructional-video events often unfold one after another.Source: §1 · p. 2
- The method generates events sequentially, conditioning each prediction on previously detected events to exploit temporal context and reduce redundancy.This reframes event detection as sequence generation and uses prior events as context.Source: §1 · pp. 2–3
- The unified video-event-text framework uses three pre-training tasks to strengthen the association between event detection and captioning.The tasks include MLM and MVFR for video-text representation, and MEFM for video-event representation.Source: §1 · pp. 2–3
- The model significantly outperforms state-of-the-art methods on both event detection and captioning on ActivityNet, generates more diverse events, and gains further from out-of-domain pre-training data.The reported comparison holds when pre-trained on ActivityNet without extra data; additional out-of-domain captioning data brings further gains.Source: §1 · p. 3
2 Related Works
Prior dense video captioning work either handled event detection and captioning separately or introduced mechanisms to connect them. The paper identifies unexplored temporal relations between events and the complexity of proposal-based detection as remaining concerns.
2 Related Works
- Earlier work split dense video captioning into event proposal generation and caption generation, with later methods improving proposal redundancy or event representations.Other approaches aimed to associate the subtasks through descriptiveness regression, a top-down pipeline, or simultaneous proposal and description generation.Source: §2 · pp. 3–4
- Inter-task approaches include bridging detection and captioning with descriptiveness regression, generating coarse sentences before aligning event fragments, and jointly producing proposals and descriptions.The top-down approach refines captions with Dual-Path Cross Attention, while another method decodes intermediate features and event queries.Source: §2 · pp. 3–4
- Prior methods do not explore temporal relations between events, and proposal-based “propose-then-select” detection relies on many candidates and is computationally complex.The paper positions sequence generation and a unified framework as its alternative.Source: §2 · p. 4
3 Method
The method places video, event, and caption representations in one multimodal pre-training framework, then fine-tunes it for event detection and captioning. Its proxy tasks model language, video features, and event boundaries.
3 Method
- The unified framework alternately pre-trains on three proxy tasks, then fine-tunes for event detection and event captioning.The model treats video, events, and captions as three modalities.Source: §3 · p. 4, §3.3 · p. 7
3.1 Model Architecture
- Independent transformers encode each modality before a cross-modal transformer captures information across modalities.Event intervals are represented as frame-level binary vectors, with positional embeddings preserving event order.Source: §3 · p. 4, §3.1 · p. 5, Figure 2 · p. 5
3.2 Proxy Tasks
- MLM predicts masked caption words, MVFR reconstructs masked video features, and MEFM predicts masked event boundaries from surrounding events and video.For MVFR, the model uses remaining video features and the event description; MEFM uses video and the surrounding event sequence.Source: §3.2 · pp. 6–7
3.3 Pre-Training and Fine-Turning for DVC Task
- Because MLM and MVFR use video, event, and text inputs while MEFM uses video and events, training alternates between the corresponding batches.The framework combines MLM and MVFR losses for three-modality batches and uses the MEFM loss for two-modality batches.Source: §3.3 · p. 7
- After pre-training, ED and EC are fine-tuned autoregressively with future items masked; inference generates events until an end-event token, then captions each event.The predicted event sequence is converted to timestamps before caption generation.Source: §3.3 · pp. 7–8
4 Experiments
Experiments compare the model with prior systems on ActivityNet, examine evaluation metrics and model components, and test additional out-of-domain pre-training data.
4.1 Experimental Settings
- Experiments use ActivityNet Captions’ official 10,009/4,925/5,044 train/validation/test split; SODA is the main dense-caption metric because Evaluator2018 scores rose under adverse event edits.The evaluation includes event-detection precision and recall, captioning metrics on ground-truth events, and dense-captioning scores on generated events.Source: §4.1 · pp. 8–10, Table 1 · p. 9
4.2 Comparison with State-of-the-art Methods
- On ActivityNet, detection average recall rose from 55.58% to 59.00% and average precision from 58.07% to 60.32%; the model also led all methods on SODAold and SODAmr.The detection results use C3D features; dense-captioning comparisons use generated event proposals.Source: Table 2 · p. 10, §4.2 · pp. 10–11, Table 3 · p. 11
4.3 Ablation Studies
- Adding MEFM raised captioning BLEU@4 from 2.38 to 2.59, METEOR from 10.48 to 10.94, and CIDEr from 51.08 to 52.13.These captioning ablations use ground-truth events to isolate caption-generation quality.Source: Table 4 · p. 12, §4.3 · p. 12
- MLM and MVFR raised detection average recall from 56.49 to 58.32 and average precision from 58.12 to 59.95; CPT raised them to 59.00 and 60.32.The best trade-off λ was 1/2 for captioning and 1/3 for detection.Source: Table 5 · p. 12, §4.3 · pp. 12–13, Figure 3 · p. 13
4.4 Pre-training with Out-of-domain Data
- With 676K extra MSRVTT, TGIF and VATEX video-caption pairs, BLEU@4, METEOR and CIDEr gains exceeded 0.4, 0.8 and 4 points respectively.Because those datasets lack event annotations, pre-training on them uses MLM and MVFR before fine-tuning on ActivityNet Captions.Source: §4.4 · pp. 13–14, Figure 4 · p. 14
5 Conclusion
The model treats event detection as sequence generation to exploit temporal relationships and unifies detection and captioning through pre-training and fine-tuning. On ActivityNet Captions, it outperforms state-of-the-art methods on both tasks, with additional gains from more out-of-domain data.
5 Conclusion
- The model significantly outperforms state-of-the-art methods on both event detection and event captioning, with additional gains from more out-of-domain data.The evaluation is on the ActivityNet Captions dataset.Source: §5 · p. 14
- Event detection is defined as sequence generation to exploit temporal relationships between events for more accurate and diverse detection.Source: §5 · p. 14
- A unified pre-training and fine-tuning framework introduces an event modality and three pre-training tasks to naturally connect detection and captioning.Source: §5 · p. 14