Source-linked AI summary
Dense-Captioning Events in Videos
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, Juan Carlos Niebles
TL;DR
The paper addresses the problem of detecting and describing multiple, temporally localized events in videos, including events with different durations and overlaps. It proposes single-pass multi-scale event proposals and contextual captioning, and introduces ActivityNet Captions to benchmark the task. The model is demonstrated on dense-captioning events, video retrieval, and event localization.
Problem
Existing video models often focus on limited action categories or a primary event, while dense-captioning events requires describing and temporally localizing multiple events.
Method
The model combines a single-pass multi-scale proposal module with a captioning module that attends over past and future events.
Results
The paper reports results for dense-captioning events, video retrieval, and event localization, including detection and description in long or streaming videos.
Takeaways & Limitations
ActivityNet Captions provides a benchmark of temporally localized, potentially overlapping event descriptions for evaluating dense video understanding.
Takeaways & Limitations
When events have high temporal overlap, the model may fail to distinguish them and repeat captions.
Abstract
from arXiv · showhide
Most natural videos contain numerous events. For example, in a video of a "man playing a piano", the video might also contain "another man dancing" or "a crowd clapping". We introduce the task of dense-captioning events, which involves both detecting and describing events in a video. We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language. Our model introduces a variant of an existing proposal module that is designed to capture both short as well as long events that span minutes. To capture the dependencies between the events in a video, our model introduces a new captioning module that uses contextual information from past and future events to jointly describe all events. We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events. ActivityNet Captions contains 20k videos amounting to 849 video hours with 100k total descriptions, each with it's unique start and end time. Finally, we report performances of our model for dense-captioning events, video retrieval and localization.
1. Introduction
Dense-captioning events addresses the need to detect and describe multiple temporally localized events, including overlapping events across different time scales. The paper proposes single-pass proposal and contextual captioning modules and introduces ActivityNet Captions for evaluation.
- Dense-captioning events requires generating descriptions for multiple video events while localizing each event with independent start and end times.
- Events can range from seconds to minutes and may overlap, requiring detection across multiple temporal scales.The paper contrasts long piano recitals with brief applause and notes that events may occur concurrently.
- The proposal module detects short and long events in a single forward pass, avoiding repeated processing over temporal sliding windows.It extends action-proposal methods to multi-scale event detection.
- The captioning module uses context from past and future events to jointly describe events and can improve caption generation.An online variant attends only to past events for streaming video.
- ActivityNet Captions contains 20k videos with temporally localized descriptions, including videos up to 10 minutes and overlapping events.Each video has 3.65 sentences on average, and descriptions refer to unique segments.
- The model reports results for dense-captioning events, video retrieval, and event localization, including detection and description in long or streaming videos.
2. Related work
Dense-captioning events connects temporal action proposals with video captioning while addressing limitations in prior approaches. It adds single-pass multi-scale localization, temporal context, and a benchmark containing overlapping, time-localized events.
- The paper bridges temporal action proposals and video captioning by jointly localizing events and describing them with natural language.
- Prior proposal methods relied on sliding windows, whereas this work outputs overlapping proposals at every time step in a single video pass.
- The proposal method extends DAPs with longer frame strides to detect events across multiple temporal scales.
- Earlier video-captioning systems often pooled frame features globally, limiting them to short clips with one major event.
- Paragraph-captioning work generated multiple sentences but did not temporally localize them and used a small, nonoverlapping cooking-video benchmark.ActivityNet Captions instead contains overlapping events with temporal annotations.
- The proposed captioning module incorporates temporal context by attending over other events in the video.This design adapts the role of spatial context in dense-image-captioning to the temporal domain.
3. Dense-captioning events model
The architecture jointly detects temporally localized events across short and long videos and describes them using context from neighboring events. It combines a single-pass proposal module with context-aware captioning.
- The system takes video frames as input and outputs temporally localized sentences, each defined by start and end times and a variable-length word sequence.
- 3.1. Event proposal module: Proposals include event times, confidence scores, and hidden representations; only proposals above a score threshold are passed to the captioning model.The hidden representation of each proposal becomes an input to language generation.
- 3.1. Event proposal module: C3D features feed a multi-scale proposal module that predicts short and long events in a single forward pass.The proposal module is designed to avoid densely applying the language model over sliding windows during inference.
- 3.1. Event proposal module: The proposal module retains overlapping event proposals as separate events rather than removing them with non-maximum suppression.
- 3.2. Captioning module with context: For each event, attention-weighted hidden representations from past and future proposals are concatenated with its own representation before captioning.The resulting context is supplied to an LSTM so descriptions can incorporate events that happened earlier or will happen later.
- 3.3. Implementation details.: Training uses separate proposal and captioning losses, alternates their optimization, and does not fine-tune the C3D feature extractor.The captioning and proposal losses are weighted with λ1 = 1.0 and λ2 = 0.1.
4. ActivityNet Captions dataset
ActivityNet Captions links videos to temporally localized sentences describing events, including events across varying durations and simultaneous events. Its annotations cover major video actions, emphasize action-centric language, and show substantial agreement between annotators.
- Each ActivityNet Captions video contains an average of 3.65 temporally localized sentences, producing 100k sentences across 20k videos.
- 10% of temporal descriptions overlap, indicating that annotations capture simultaneous events.
- Each sentence describes 36 seconds and 31% of its video, while each video paragraph covers 94.6% of the full video.
- ActivityNet Captions uses more verbs and pronouns than Visual Genome, shifting descriptions toward actions and references to entities in prior sentences.
- Annotators show 70.2% average temporal intersection-over-union agreement with maximally overlapping sentences from independently written paragraphs.
5. Experiments
The experiments evaluate dense-captioning, localization, retrieval, and captioning variants, showing that contextual event information improves captioning and retrieval while multi-stride proposals support event localization across timescales.
- Dense-captioning events: The model is evaluated on ActivityNet Captions for detecting multiple events and describing each with natural language.The joint metric measures captioning and localization using average precision across tIoU thresholds of 0.3, 0.5, and 0.7.
- Dense-captioning events: Past context improves captioning of subsequent events in the online model, while future context further improves performance in the full model.Table 2 reports effects on the first, second, and third events in a video.
- Localization: Sampling videos at varying strides improves proposal-module localization, especially for longer events.The tested strides are 1, 2, 4, and 8, and each stride can be computed in parallel in a single pass.
- Dense-captioning events: 22.19 and 22.94 CIDEr scores result from adding past-event context, compared with 20.97 for S2VT and the no-context model.The online-attn and online variants use past context, while the full models additionally use future context.
- Dense-captioning events: 24.24 and 24.56 CIDEr scores result when future-event context is added through the full-attn and full models.Attention yields larger improvements among videos containing more events, suggesting greater usefulness for longer videos.
- Video and paragraph retrieval: Contextual encoding improves video and paragraph retrieval, which evaluates matching sentences and videos in both directions.The retrieval model combines contextual sentence representations with multiple video proposals and is trained using a max-margin loss.
- Dense-captioning events: When proposed segments have high overlap, the model can fail to distinguish events and repeat captions.This is an observed limitation of event separation in dense-captioning.
6. Conclusion
The paper introduces dense-captioning events and addresses short-to-long, related video events with multi-scale proposals and contextual captioning. It releases ActivityNet Captions and demonstrates gains in captioning, retrieval, and localization.
- Conclusion: Dense-captioning events requires detecting multiple temporally localized video events and describing each with natural language.The task addresses events ranging from less than a second to several minutes and events that are related to one another.
- Conclusion: The proposal module samples frames at different strides to propose events at multiple time scales in one pass.The captioning module attends over neighboring events to use their context when generating captions.
- Conclusion: Context improves captioning and video retrieval, while different sampling strides improve event localization.The paper also releases ActivityNet Captions for dense-captioning events.
- Conclusion: ActivityNet Captions contains videos whose paragraphs average 3.65 sentences.The dataset passage reports the distribution of sentences within paragraphs.
7. Supplementary material
The supplementary material characterizes ActivityNet Captions as a large, temporally localized dataset spanning long videos, overlapping events, and varied natural-language descriptions. It also details annotation procedures, worker behavior, and dataset coverage across video semantics.
- Dataset statistics: ActivityNet Captions contains videos averaging 180 seconds, with the longest exceeding 10 minutes, and 100k temporally localized sentences.The dataset contains two orders of magnitude more videos than TACoS multi-level and supports an open-domain setting.
- Dataset statistics: Each video contains 3.65 ± 1.79 sentences on average, while each sentence averages 13.48 ± 6.33 words.Each video contains 40 ± 26 words on average.
- Temporal coverage: Each additional minute of video adds approximately one sentence description, while annotations concentrate nearer the middle than the beginning or end.The number of sentences is positively correlated with video length.
- Semantic coverage: ActivityNet Captions generally describes people and their actions, with entry-level categories represented more consistently than nuanced categories.Categories such as brushing hair and playing violin are well represented, whereas powerbocking and cumbia are less common.
- Annotation process: Workers first wrote paragraphs describing major events, then labeled each sentence with its start and end timestamps.Workers were instructed to write at least three sentences, with each sentence at least five words long and coherent as part of the paragraph.
- Annotation process: Workers viewed videos at an average rate of 1.91X, while annotation took an average of 2.88 times the video duration in seconds.The median viewing rate was 1X, and the median annotation time was 1.46 times the video duration.