PaperlayerView exact revision on arXiv

Source-linked AI summary

Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning

Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, Yong Xu

arXiv:1804.00100v2cs.CV

TL;DR

Dense video captioning requires localizing and describing all events, but prior methods neglect future context and cannot distinguish events ending at the same time. The paper uses bidirectional proposals, attentive event-content fusion with context gating, and a coupled proposal-captioning framework, reporting superior localization and description performance.

  • Problem

    Dense video captioning must localize and describe multiple events in long videos, while prior methods omit future context and represent same-time-ending proposals with one hidden state.

  • Method

    The method combines Bidirectional SST, attentive fusion of proposal states and video content with dynamic context gating, and joint ranking in a unified end-to-end framework.

  • Results

    The experiments demonstrate superiority in both localizing events and describing them.

  • Takeaways & Limitations

    Bidirectional context and fused event representations support the paper’s unified approach to dense event localization and description.

Abstract

from arXiv · show

Dense video captioning is a newly emerging task that aims at both localizing and describing all events in a video. We identify and tackle two challenges on this task, namely, (1) how to utilize both past and future contexts for accurate event proposal predictions, and (2) how to construct informative input to the decoder for generating natural event descriptions. First, previous works predominantly generate temporal event proposals in the forward direction, which neglects future video context. We propose a bidirectional proposal method that effectively exploits both past and future contexts to make proposal predictions. Second, different events ending at (nearly) the same time are indistinguishable in the previous works, resulting in the same captions. We solve this problem by representing each event with an attentive fusion of hidden states from the proposal module and video contents (e.g., C3D features). We further propose a novel context gating mechanism to balance the contributions from the current event and its surrounding contexts dynamically. We empirically show that our attentively fused event representation is superior to the proposal hidden states or video contents alone. By coupling proposal and captioning modules into one unified framework, our model outperforms the state-of-the-arts on the ActivityNet Captions dataset with a relative gain of over 100% (Meteor score increases from 4.82 to 9.65).

1. Introduction

Dense video captioning must localize and describe multiple events in long, untrimmed videos. This paper addresses missing future context in proposal generation and weak discrimination among overlapping proposals.

  • 1. Introduction: Forward-only proposal processing neglects future context that may be closely related to the current event.Figure 1 contrasts a current hair-drying proposal with a future hair-molding context.
  • 1. Introduction: Dense video captioning simultaneously localizes events and generates descriptions for them in long, complicated video sequences.ActivityNet Captions videos are described as approximately 120 seconds long, compared with approximately 20-second clips in conventional video captioning.
  • 1. Introduction: Sliding-window proposal methods cannot produce proposals longer than the predefined window, motivating SST’s single-pass dense proposal predictions.SST predicts proposals ending at each time step with k different offsets.
  • 1. Introduction: Using one proposal-module hidden state for all proposals ending at the same time step prevents those events from being distinguished.The paper gives P1 and P2 as examples that both receive the same hidden state h_t.
  • 1. Introduction: The paper proposes Bidirectional SST, attentive fusion of proposal states with video content, and joint ranking of proposal-caption pairs.The contributions combine past, current, and future context, discriminative event representations, and confidence-aware inference selection.

2. Related Work

Prior work separates temporal proposal generation from video or paragraph captioning, while dense video captioning automatically grounds multiple sentences to events. The proposed framework couples bidirectional proposal encoding with caption generation.

  • 2. Related Work: Temporal action proposal methods generate candidate windows using approaches including dictionary learning, 3D CNNs, clip pyramids, and LSTMs.These methods differ in whether they generate class-independent, multi-scale, jointly refined, or sequence-encoded proposals.
  • 2. Related Work: Video captioning research has progressed from template-based descriptions to recurrent, translation-based, embedding-based, and attention-based sequence models.The cited approaches model semantic relationships between visual components, video and language, or temporal visual content.
  • 2. Related Work: The proposed system couples proposal and captioning modules and trains them end-to-end.Its framework encodes visual features bidirectionally, feeds encoder states to proposal prediction, and uses a decoder LSTM for sentences.
  • 2. Related Work: Video paragraph generation produces multiple fluent sentences and models cross-sentence and word dependencies, but relies on ground-truth event alignment at test time.Dense video captioning removes that alignment requirement by automatically grounding multiple sentences to temporal locations.
  • 2. Related Work: Dense video captioning combines automatic event grounding with multiple-sentence generation, making it more challenging than paragraph generation.The related-work discussion identifies ActivityNet Captions as a dataset introduced with an earlier dense-captioning model.
  • 2. Related Work: Compared with the earlier dense-captioning model, this method uses both past and future context for proposals instead of only past context.The earlier model uses a multi-scale DAPs proposal module and LSTM hidden states to represent detected events.

3. Method

The method couples bidirectional proposal generation with attentive event-context fusion, context gating, caption decoding, and joint proposal-caption ranking in an end-to-end framework.

  • Proposal Module: The proposal module represents each candidate using predefined temporal anchors ending at the current time step.Each anchor has a predefined length, and classifiers produce confidence scores for the corresponding proposal intervals.
  • Proposal Module: Bidirectional SST encodes past, current, and future contexts to generate temporal event proposals.A forward LSTM processes past visual clues, while a reverse pass captures future context; proposal scores from both passes are fused before threshold-based selection.
  • Captioning Module: The decoder receives fused proposal hidden states and detected event clip features rather than proposal hidden states alone.The event clip features are C3D representations, while forward and backward proposal states serve as context vectors encoding past and future information.
  • Captioning Module: Temporal dynamic attention weights event clip features during decoding to construct visual input for the captioning LSTM.The attended visual feature is generated as a weighted sum, with attention conditioned on the proposal and decoder state.
  • Captioning Module: Context gating dynamically balances the attended event feature and surrounding context when generating each word.The gate uses the word embedding and previous decoder state to measure how much surrounding context should contribute at the current decoding stage.
  • Inference and Training: Joint ranking selects proposal-caption pairs using both proposal scores and caption confidence during inference.The proposal and captioning modules are coupled and trained with proposal and captioning losses.

4. Experiment

Experiments evaluate proposal localization, captioning with ground-truth proposals, and complete dense captioning on ActivityNet Captions and THUMOS-14. Bidirectional proposals, fused event and context representations, attention, context gating, and joint ranking consistently improve reported performance, while very long events remain challenging.

  • Experimental Tasks: The experiments evaluate event localization, captioning with ground-truth proposals, and the complete dense captioning system.Event localization measures proposal quality, captioning measures the captioning module, and dense captioning measures the whole system.
  • Event Localization: Bidirectional SST surpasses Random and Forward/Backward SST with clear margins on ActivityNet Captions proposal evaluation.The comparison uses F1 across different tIoU thresholds because Random achieves high recall but very low precision.
  • Event Localization: Bidirectional prediction improves proposal quality over single-direction prediction by incorporating future context.On THUMOS-14, Bi-SST also outperforms SST and Backward SST, especially with smaller proposal numbers, and achieves new state-of-the-art results.
  • Dense Event Captioning: All six evaluated variants outperform the baseline method by large margins on ActivityNet Captions validation captioning.The baseline is SST + H, which represents proposals using SST hidden states and context vectors.
  • Dense Event Captioning: Combining event clip features with context vectors outperforms either representation alone, while temporal dynamic attention further improves all scores.The comparison primarily uses Meteor because it is reported to align better with human judgments when only one reference sentence is available; other metrics show slight inconsistency.
  • Dense Event Captioning: Context gating and joint ranking further improve captioning and confidence in proposal-caption pairs.Context gating models the relative contributions of event features and contexts during decoding, while joint ranking considers proposal and caption confidence together.
  • Qualitative and Length Analysis: Event clip fusion helps the system attend to the current event while referring to context, reducing incorrect or semantically ambiguous descriptions in qualitative examples.Without event clip fusion, the system tends to make more semantic mistakes.
  • Qualitative and Length Analysis: Performance degenerates for proposals longer than 60s, indicating that understanding and describing long events remains challenging.Bi-SST+H performs better than SST-H as proposal length varies, but both systems decline steeply for very long proposals.

5. Conclusion

The paper addresses context fusion and event representation through a bidirectional, end-to-end framework, with experiments showing superiority in both event localization and description.

  • The framework handles context fusion and event representation as the two central dense video captioning challenges.
  • Bidirectional SST encodes past and future contexts to better localize current events.The proposal and captioning modules are coupled for end-to-end training.
  • Extensive quantitative and qualitative experiments demonstrate superiority in both localizing events and describing them.

6. Supplementary Material

Supplementary results provide qualitative examples and category-level performance for the best dense captioning model, revealing substantial variation across activity categories.

  • Figures 6 and 7 show qualitative results from the best model, Bi-SST+E+H+TDA+CG+Ranking.The generated captions often contain more details than the ground truths.
  • The best-performing category is Tennis serve with ball bouncing, with Meteor: 15.1.Other top categories include Skiing at Meteor: 14.7 and Calf roping at Meteor: 14.3.
  • The worst-performing category is Having an ice cream, with Meteor: 5.3.Other low-performing categories include Doing Karate at Meteor: 5.4 and Doing a powerbomb at Meteor: 6.3.

GT Ours

The supplementary examples pair ground-truth and generated captions across lawn mowing, water sports, indoor soccer, skateboarding, tennis, and surfing, with selected proposal tIoUs reported.

  • The examples include lawn-mowing captions describing a man mowing or continuing to move a machine across a yard.
  • Tennis and surfing examples report tIoUs of 97.7% and 98.2% for tennis, and 99.4% and 96.5% for surfing.
  • Water-sports examples cover surfing, kayaking, rafting, and rough-water scenes, with reported proposal tIoUs from 69.75 to 99.6%.
  • Skateboarding examples depict riding, tricks, falling, and laughing, with reported tIoUs of 98.7%, 99.8%, and 42.0%.
Loading 1804.00100v2…