PaperlayerView exact revision on arXiv

Source-linked AI summary

Streamlined Dense Video Captioning

Jonghwan Mun, Linjie Yang, Zhou Ren, Ning Xu, Bohyung Han

arXiv:1904.03870v1cs.CV

TL;DR

Dense video captioning requires coherent descriptions of multiple temporally dependent events, whereas many existing methods caption detected proposals independently. The paper detects event sequences, generates captions sequentially with visual and linguistic context, and trains the captioning network with event- and episode-level reinforcement-learning rewards. It achieves state-of-the-art accuracy on ActivityNet Captions in METEOR.

  • Problem

    Dense video captioning must describe interdependent events coherently, but existing approaches often detect proposals and caption them without modeling temporal dependency across events.

  • Method

    The framework adaptively selects ordered event proposals and generates sequential captions conditioned on preceding events, using reinforcement learning with event- and episode-level rewards.

  • Results

    State-of-the-art accuracy is achieved on the ActivityNet Captions dataset in terms of METEOR.

  • Takeaways & Limitations

    Modeling temporal dependency and visual and linguistic context supports coherent caption generation across events in a video episode.

Abstract

from arXiv · show

Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events. Most existing approaches handle this problem by first detecting event proposals from a video and then captioning on a subset of the proposals. As a result, the generated sentences are prone to be redundant or inconsistent since they fail to consider temporal dependency between events. To tackle this challenge, we propose a novel dense video captioning framework, which models temporal dependency across events in a video explicitly and leverages visual and linguistic context from prior events for coherent storytelling. This objective is achieved by 1) integrating an event sequence generation network to select a sequence of event proposals adaptively, and 2) feeding the sequence of event proposals to our sequential video captioning network, which is trained by reinforcement learning with two-level rewards at both event and episode levels for better context modeling. The proposed technique achieves outstanding performances on ActivityNet Captions dataset in most metrics.

1. Introduction

Dense video captioning must describe multiple interdependent events coherently, but existing proposal-based methods caption selected events independently. The proposed framework detects event sequences, conditions captions on prior events, and uses two-level reinforcement-learning rewards to improve coherence and event description quality.

  • Motivation: Dense video captioning extends single-sentence captioning to multiple interdependent events forming a video-level episode.A single sentence may be insufficient when videos contain several events connected within one story.
  • Motivation: Existing approaches detect event proposals and generate captions for selected proposals independently, limiting temporal dependency modeling.Their event proposal and captioning networks operate as separate stages, with captions generated independently for detected events.
  • Approach: The framework detects an ordered sequence of event proposals and generates captions sequentially, conditioning each caption on prior events and captions.An event sequence generation network adaptively selects ordered events from candidate proposals before sequential caption generation.
  • Training: Two-level reinforcement-learning rewards target precise event descriptions and coherence across the episode.Event-level rewards capture specific content, while episode-level rewards encourage all generated captions to form a coherent story.
  • Results: The algorithm achieves state-of-the-art performance on ActivityNet Captions with large margins over methods using the existing framework.The supplied introduction reports this outcome without specifying metric values.

2. Related Work

Prior video captioning methods mainly describe short, single-event videos or generate captions independently for detected proposals. The proposed approach instead selects representative event sequences and exploits visual and linguistic context across events for coherent, comprehensive captions.

  • Single- and paragraph-level captioning: Earlier video captioning methods commonly encode videos and decode single natural-language sentences, restricting them to short videos with one event.Later work extends this paradigm to paragraph generation for long videos.
  • Dense video captioning: Dense video captioning typically separates event detection from caption generation, producing one caption per proposal independently.Final proposal selection can also depend on manual thresholding strategies.
  • Dense video captioning: Existing dense captioning methods incorporate visual context or proposal-quality mechanisms but still generally apply captioning independently to event proposals.The related approaches include attention, bidirectional RNNs, context gating, temporal regressions, and attribute-augmented captioning.
  • Proposed distinction: The proposed method selects a small set of representative event sequences for sequential caption generation using visual and linguistic context across selected events.This design is presented as a contrast to prior methods that do not exploit linguistic context between independently captioned proposals.

3. Our Framework

The framework detects an ordered event sequence from candidate proposals and generates captions sequentially with visual and linguistic context from preceding events. Its components combine proposal extraction, adaptive sequence selection, hierarchical captioning, and reinforcement learning with event- and episode-level rewards.

  • Overall Framework: The framework extracts candidate event proposals, selects an ordered event sequence, and generates captions conditioned on preceding events and captions.The three models are trained supervised, while the Sequential Captioning Network is additionally optimized with reinforcement learning.
  • Training: Reinforcement learning applies event-level rewards for precise event content and episode-level rewards for coherence across the generated story.The captioning network is optimized using both reward levels after supervised training.
  • Event Proposal Network: The Event Proposal Network uses SST to generate candidate proposals and proposal representations from temporal start and end states.SST scans fixed-length video segments, retains top proposals, and applies non-maximum suppression to obtain the candidate set.
  • Event Sequence Generation Network: The Event Sequence Generation Network uses a Pointer Network to select an ordered subset of proposals adaptively until an END proposal terminates the sequence.Proposal selection uses representations encoded in increasing starting-time order and likelihoods over available candidates.
  • Event Sequence Generation Network: The sequence generator selects 2.85 events on average, compared with 3.65 average events per ActivityNet Captions video.This contrasts with existing approaches that may produce at least 50 detected events through manual thresholding.
  • Sequential Captioning Network: The Sequential Captioning Network uses episode and event RNNs so each event caption is generated from the current episode context and returned to update it.The scheme exploits visual and linguistic context across events, unlike independent caption generation that lacks linguistic dependency.

4. Training

Training combines supervised learning for event proposals, event-sequence selection, and sequential captioning with reinforcement learning for the captioning network. The reinforcement objective uses event- and episode-level rewards to improve individual descriptions and story coherence.

  • Training procedure: The event proposal network is learned first, then fixed while the event sequence generation and sequential captioning networks are trained.The latter two networks use supervised training before reinforcement learning further optimizes captioning.
  • Supervised learning: ESGN selects proposals that highly overlap with ground-truth events by minimizing a binary cross-entropy objective over proposal-selection likelihoods.The target is the likelihood that proposal m is selected as event n, with temporal IoU defining proposal overlap.
  • Supervised learning: The sequential captioning network learns from ground-truth event sequences and descriptions using teacher forcing and word-level negative log-likelihood.The episode and event RNNs receive ground-truth events and captions as inputs, respectively.
  • Reinforcement learning: Reinforcement learning minimizes negative expected rewards for sampled descriptions generated from detected event sequences.The sampled descriptions are associated with detected sequences containing Ns events, and each description receives an individual reward.
  • Reinforcement learning: Two-level rewards evaluate the current event and the whole episode, using caption-similarity scores against descriptions generated from reference event proposals.The event-level and episode-level terms encourage sampled captions whose scores exceed those from the reference event sequence; METEOR and CIDEr define the similarity function.

5. Experiments

Experiments on ActivityNet Captions evaluate event detection, dense captioning, ablations, and qualitative coherence. SDVC selects compact event sequences, models context sequentially, and improves reported performance over comparison systems and variants.

  • Dataset and evaluation: ActivityNet Captions contains 20k YouTube videos averaging 120 seconds, with 3.65 localized events and 13.48-word descriptions per video.
  • Dataset and evaluation: Evaluation averages recall, precision, METEOR, CIDEr, and BLEU across tIoU thresholds of 0.3, 0.5, 0.7, and 0.9.
  • Event detection: ESGN outperforms MFT on most event-detection tIoUs, especially recall, validating the proposed event-sequence selection algorithm.
  • Comparison with other methods: SDVC achieves stronger caption quality than comparison methods while using only 2.85 average proposals, versus 97.61–1,000 for several alternatives.
  • Comparison with other methods: SDVC achieves competitive test performance using only basic visual features, whereas other methods use additional modalities or ensemble models.
  • Ablation studies: Ablations show that ESGN reduces proposals from 77.99 to 2.85, sequential context improves over independent captioning, and reinforcement learning further improves caption quality.
  • Ablation studies: Training with both event-level and episode-level rewards is effective according to the reward-level ablation.
  • Qualitative results: Qualitative examples show SDVC preserves linguistic references such as “they” and temporal continuity such as “continue,” unlike independent captions.

6. Conclusion

The conclusion presents SDVC as a framework that explicitly models temporal dependencies across video events. It combines adaptive event-sequence selection, sequential captioning, and two-level reinforcement learning, achieving state-of-the-art METEOR accuracy on ActivityNet Captions.

  • SDVC models temporal dependency across events while using visual and linguistic context for coherent caption generation.
  • The event sequence generation network adaptively detects ordered event proposals, after which captions condition on preceding events.
  • The captioning network is trained supervised and further optimized with event-level and episode-level reinforcement-learning rewards.
  • SDVC achieves state-of-the-art accuracy on ActivityNet Captions in METEOR.

A. Details of Event RNN

The event RNN generates each event description from event features and episode context. Temporal attention selects segment information, while gated features support word prediction.

  • The event RNN generates a description from an event and context, returning features for caption generation.
  • C3D(e) supplies segment-level descriptors and Vis(e) supplies the proposal network’s visual representation.
  • The event RNN initializes its hidden state with the episode RNN context and applies Temporal Dynamic Attention at each word step.
  • Temporal Dynamic Attention produces an attentive segment vector, while caption generation computes gated event features for vocabulary prediction.
  • The gating operation uses a sigmoid, vector concatenation, and element-wise multiplication to combine event information.

B. Visualization of Event Selection

Figure 5 visualizes how the event sequence generation network selects event proposals from candidates. The selected proposals highly overlap with ground-truth events.

  • The visualization compares selected event proposals with candidate proposals and ground-truth events.
  • The selected proposals highly overlap with the ground-truth events, indicating successful event selection.
Loading 1904.03870v1…