PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Video Captioning using Graph-based Sentence Summarization

Zhiwang Zhang, Dong Xu, Wanli Ouyang, Luping Zhou

arXiv:2506.20583v1cs.CVcs.AI

TL;DR

Dense video captioning methods do not sufficiently model scene evolution within long event proposals, especially when their scenes and objects change. GPaS partitions proposals into segments and summarizes their visual and semantic information with graph-based word relationships integrated with LSTM. The paper reports that GPaS outperforms several ablated alternatives and evaluates it on ActivityNet Captions and YouCook II.

  • Problem

    Existing methods do not sufficiently explore scene evolution within event proposals, limiting performance when scenes and objects change substantially over a long proposal.

    Source: §I · p. 1

  • Method

    GPaS partitions event proposals into segments, then summarizes their visual and semantic information using graph-based word relationships coupled with LSTM.

    Source: §I · p. 1

  • Results

    GPaS outperforms TA, PM-ave, and PM-best, while ablations report higher Meteor scores for expanded than basic graphs in both tested GLI variants.

    Source: E. Ablation Study · p. 9

  • Takeaways & Limitations

    The reported gain from the expanded graph indicates that exploiting different semantic levels can produce better dense video captions.

    Source: §V · p. 10

Abstract

from arXiv · show

Recently, dense video captioning has made attractive progress in detecting and captioning all events in a long untrimmed video. Despite promising results were achieved, most existing methods do not sufficiently explore the scene evolution within an event temporal proposal for captioning, and therefore perform less satisfactorily when the scenes and objects change over a relatively long proposal. To address this problem, we propose a graph-based partition-and-summarization (GPaS) framework for dense video captioning within two stages. For the ``partition" stage, a whole event proposal is split into short video segments for captioning at a finer level. For the ``summarization" stage, the generated sentences carrying rich description information for each segment are summarized into one sentence to describe the whole event. We particularly focus on the ``summarization" stage, and propose a framework that effectively exploits the relationship between semantic words for summarization. We achieve this goal by treating semantic words as nodes in a graph and learning their interactions by coupling Graph Convolutional Network (GCN) and Long Short Term Memory (LSTM), with the aid of visual cues. Two schemes of GCN-LSTM Interaction (GLI) modules are proposed for seamless integration of GCN and LSTM. The effectiveness of our approach is demonstrated via an extensive comparison with the state-of-the-arts methods on the two benchmarks ActivityNet Captions dataset and YouCook II dataset.

I. INTRODUCTION

Dense video captioning must localize events and describe them, but proposal-level captioning can miss scene evolution when scenes or objects change within long proposals. GPaS partitions proposals into segments, then summarizes their visual and semantic information using word relationships; its related work builds on temporal proposals and video-captioning approaches while distinguishing its graph-based summarization.

I. INTRODUCTION

  • The introduction identifies temporal proposal generation and event captioning as the two sub-tasks of dense video captioning.It describes the task as detecting events and giving language descriptions in long untrimmed videos.Source: §I · p. 1
  • Existing methods caption whole proposals without sufficiently modeling scene evolution, and perform less well when scenes and objects change substantially within a long proposal.Source: §I · p. 1
  • GPaS partitions each proposal into segments, generates segment descriptions, and summarizes their visual features and words into one event sentence using a graph to model word relationships.The method focuses on the video-captioning subtask.Source: §I · p. 1

A. Temporal action proposals

  • Temporal proposal methods include DAPs, SST, and its bidirectional extension, which use visual context to localize events in untrimmed video.Temporal proposals are segments that may contain actions or events.Source: A. Temporal action proposals · p. 2

B. Image and video captioning

  • Prior partition-and-summarization approaches used representative-sentence selection, sequence-to-sequence integration, or hierarchical attention; GPaS instead uses GCN to model word relationships.The cited approaches are,, and; LexRank is identified as.Source: B. Image and video captioning · p. 2

C. Graph Convolutional Network and Recurrent Neural Network

GPaS represents semantic words as graph nodes and tightly couples GCN with LSTM, while using segment-level sentences to model multiple semantic levels. Its pipeline partitions proposals into segments, extracts visual features, and generates a description for each segment.

C. Graph Convolutional Network and Recurrent Neural Network

  • GPaS tightly couples GCN nodes with LSTM cells to refine hidden representations and cell states, and uses an expanded graph to represent multiple semantic levels.Unlike, which treats detected objects as nodes and separates GCN encoding from LSTM decoding, GPaS uses semantic words as nodes.Source: C. Graph Convolutional Network and… · p. 3

III. METHODOLOGY

  • The framework partitions each event proposal into segments, extracts visual features, and generates a sentence for each segment before summarizing the proposal.The overview identifies partition and GCN-LSTM summarization as the two modules.Source: §III · p. 3, Figure 1 · p. 3

A. Partition module

  • The partition module uniformly samples 20 segments per proposal, each with 16 frames, and produces visual features and segment-level sentences.Segment sentences are generated using the captioning method in.Source: A. Partition module · p. 3
  • To make segment sentences a fixed length for training, longer captions are truncated to Lk words and shorter ones are padded with blanks.Source: A. Partition module · p. 3

B. Basic summarization module

The basic summarization module uses an encoder-decoder to turn partition words into one event-proposal sentence. Its LSTM units receive a joint visual-textual feature, with visual features attended to using the hidden state.

B. Basic summarization module

1) Basic Unit:

  • Each basic unit feeds the LSTM a joint visual-textual feature rather than semantic words alone.The feature is the output of the TVJ module.Source: 1) Basic Unit · p. 4
  • The TVJ module fuses the word embedding with visual features attended according to their importance, using the LSTM hidden state to compute attention weights.A multilayer perceptron fuses the features; softmax produces the attention weights.Source: 1) Basic Unit · p. 4

2) Implementing Encoder-word layer and Decoder using the Basic Unit:

The encoder-word layer fuses input words with visual features, while the decoder uses hidden-state and visual inputs to produce the event description.

2) Implementing Encoder-word layer and Decoder using the Basic Unit:

  • The encoder-word layer has Lm × Lk basic units, one for each segment word, and fuses input words with visual features through the TVJ module.Its final hidden representation is fed to the decoder.Source: 2) Implementing Encoder-word layer and… · p. 4
  • The encoder’s hidden representation and partition-module words provide inputs for the encoder-word layer, while decoder hidden states correspond to successive output words.The output word is obtained by transforming a decoder hidden state.Source: 2) Implementing Encoder-word layer and… · p. 4
  • The decoder takes the previous hidden state and visual features as input, with Lk basic units corresponding to output words in the final sentence.Its initial hidden state comes from the encoder-word layer’s last hidden representation.Source: 2) Implementing Encoder-word layer and… · p. 4

C. GCN-enhanced summarization module

The summarization module treats semantic words and, in an expanded design, segment-level sentences as graph nodes, using aGCN and LSTM to model relationships across and within semantic levels.

C. GCN-enhanced summarization module

  • Input words and generated output words become graph nodes with semantic features; the expanded graph adds segment-level sentences as nodes.The paper motivates this design by noting that the integrated input words affect the quality of the event’s one-sentence description.Source: C. GCN-enhanced summarization module · p. 4
  • The GLI module combines an LSTM cell with an aGCN node: aGCN models relationships across semantic levels, while LSTM models relationships within each level.The paper proposes two graph structures for these modules.Source: C. GCN-enhanced summarization module · p. 5
  • In the basic graph, directed edges connect encoder-word nodes to decoder output-word nodes, whose features are updated using aGCN.The input words are drawn from Lm × Lk encoder-word units, and the output words from Lk decoder units.Source: C. GCN-enhanced summarization module · p. 5
  • The expanded graph inserts segment-level sentence nodes between input-word and output-word nodes, with edges linking words to their corresponding sentence and sentences to decoder words.This extends the two-layer arrangement to encoder-word, encoder-segment, and decoder layers.Source: C. GCN-enhanced summarization module · p. 5

D. GCN outside LSTM

In aGCN-out-LSTM, LSTM hidden states serve as graph-node features and are refined through aGCN message passing across semantic levels.

D. GCN outside LSTM

  • aGCN-out-LSTM places the graph node outside the LSTM cell and uses each cell’s hidden state as its node feature.The method progressively refines nodes at each semantic level using updated features from the preceding level.Source: D. GCN outside LSTM · p. 5
  • Input-word nodes have no incoming edges in the basic graph, so their hidden states remain unchanged by aGCN updates.The input-word node feature is initialized from the encoder-word hidden state.Source: D. GCN outside LSTM · p. 5
  • In the basic graph, decoder output-word nodes receive aGCN updates from encoder-word nodes, using refined encoder hidden states and learnable weights.Each output-word node is pointed to by Lm × Lk input-word nodes.Source: D. GCN outside LSTM · p. 6
  • In the expanded graph, segment-level sentence nodes are refined from word-node features before decoder nodes are updated, and aGCN-refined LSTM hidden states support collaboration with LSTM during end-to-end training.The segment-node update uses refined encoder-word features; decoder updates use refined segment-level features.Source: D. GCN outside LSTM · p. 6

E. GCN inside LSTM

In aGCN-in-LSTM, graph-node features are cell states, which are refined through aGCN and then affect the corresponding LSTM hidden states.

E. GCN inside LSTM

  • In aGCN-in-LSTM, the graph node is placed inside the LSTM cell and its node feature is the cell state.The corresponding hidden states are updated using the refined states.Source: E. GCN inside LSTM · p. 6
  • The cell state combines forget-gated prior memory with input-gated content, and the hidden state is the output-gated tanh of that cell state.The passage identifies f_t, i_t, and o_t as the forget, input, and output gates; refining the cell state therefore refines the hidden state.Source: E. GCN inside LSTM · p. 6
  • For the expanded graph, cell states at the encoder-word, encoder-segment, and decoder levels are refined using the corresponding graph structure and learnable weights.The passage gives the expanded-graph update directly and omits the analogous derivation details.Source: E. GCN inside LSTM · p. 6

F. Other Details

Word prediction combines cross-entropy and discriminative losses, while reinforcement learning uses Meteor as its reward.

  • The decoder maps hidden representations to dictionary words by estimating their probabilities with cross-entropy loss.Source: F. Other Details · p. 7
  • A discriminative loss is added to cross-entropy to improve word prediction, with λd empirically set to 0.1.Source: F. Other Details · p. 7
  • Reinforcement learning is also applied to boost performance using Meteor as the reward.Source: F. Other Details · p. 7

G. Difference from Attention

The method refines LSTM states through graph-based word interactions rather than applying conventional soft attention to input features. The experiments include ActivityNet Captions and YouCook II, with YouCook II used to evaluate longer videos.

G. Difference from Attention

  • GCN-LSTM attention refines cell or hidden states and permits iterative graph-node updates, unlike soft attention’s weighted sum of input features.The paper attributes improved performance to the added nonlinearity.Source: G. Difference from Attention · p. 7

A. Datasets

  • ActivityNet Captions contains around 20,000 long videos and around 100,000 temporally annotated sentences, averaging 13.5 words each.Most videos have more than three annotated sentences.Source: A. Datasets · p. 7
  • YouCook II has around 2,000 videos averaging more than five minutes; evaluation uses its validation set and features from Resnet-200 and optical flow.The feature and evaluation setup follows.Source: A. Datasets · p. 7
  • Preprocessing lowercases alphabetic text and replaces words appearing fewer than three times with < UNK >.The resulting dictionaries contain 6,994 ActivityNet words and 1,269 YouCook II words.Source: A. Datasets · p. 7

B. Experimental Setup

The evaluation compares GPaS with ground-truth and automatically generated proposals, using ActivityNet Captions benchmarks. Results favor GPaS on the reported ground-truth and automatic-proposal comparisons.

B. Experimental Setup

  • The experimental setup fixes each proposal at 20 video segments, with a maximum of 25 words per segment-level or final sentence.Longer sentences are truncated and shorter ones are zero-padded; the summarization module takes 500 input words.Source: B. Experimental Setup · p. 7
  • Training uses Adam with minibatches of 32, 512-dimensional hidden states, and a dropout rate of 0.8.The cross-entropy learning rate starts at 0.0003 and decreases 1.25 times every three epochs.Source: B. Experimental Setup · p. 7

D. Performance Comparison

  • 11.04% Meteor on ActivityNet Captions validation with ground-truth proposals is GPaS’s highest score, 0.71% above DVC, and GPaS also leads CIDEr-D.Source: Table I · p. 7, D. Performance Comparison · p. 8
  • GPaS achieves the highest scores across all reported metrics under both evaluation criteria with automatically generated ActivityNet proposals.The comparison includes Bi-AFCG, MT, and DVC; Table II separates old-criterion results from new-criterion results.Source: Table II · p. 8, D. Performance Comparison · p. 8
  • Table III covers ActivityNet Captions testing, while Table IV covers YouCook II validation.Source: Table III · p. 8, Table IV · p. 8

E. Ablation Study

The ablations compare graph configurations, visual cues, partition-and-summarization variants, and additions to the final GPaS model. Results support expanded graphs, visual cues, and the final model’s added components.

E. Ablation Study

  • Both GCN-enhanced summarizers outperform PaS-basic, and the expanded graph places segment-level sentences between input and output words.Source: E. Ablation Study · p. 8
  • Fusing visual features with semantic words raises Meteor by 0.3% compared with PaS-basic without TVJ.Source: E. Ablation Study · p. 9
  • GPaS outperforms TA, PM-ave, and PM-best, which respectively omit partition-and-summarization, summarization, or selection beyond a representative segment sentence.Source: E. Ablation Study · pp. 8–9
  • GPaS outperforms aGCN-in-LSTM-expanded, supporting the use of reinforcement learning and confidence scoring for dense video captioning.Source: E. Ablation Study · p. 9
  • Meteor decreases without discriminative loss, while performance is relatively stable across certain ranges of Lm, Lk, and λd.Source: E. Ablation Study · p. 9

F. Qualitative Analysis

Qualitative examples show the proposed method producing reasonable captions with automatically generated proposals and better descriptions than TA despite errors in some segment-level sentences. The conclusion attributes its approach to graph-based sentence summarization, where GCN-LSTM coupling and expanded semantic graphs improve sentence representations and descriptions.

F. Qualitative Analysis

  • The method produces a better sentence description than TA even when some of its segment-level sentences are incorrect.The example uses Lm=20 segment-level sentences as input to the GCN-enhanced summarization module.Source: F. Qualitative Analysis · p. 9, Figure 7 · p. 10
  • Failure examples include a kite crosslet recognized as a person and a duplicated swimming statement that makes the caption redundant.The authors suggest reliable object detection and a penalty against repeated statements as future work.Source: F. Qualitative Analysis · pp. 9–10
  • The method generates reasonable sentences with automatically generated proposals, with proposal IoU reported alongside each example.Correct semantic words are highlighted in Fig. 6.Source: F. Qualitative Analysis · p. 9, Figure 6 · p. 10

V. CONCLUSION

  • The GCN-enhanced summarizer treats semantic words as graph nodes and couples GCN with LSTM to refine sentence representations.The authors report better message transfer between LSTM cells when GCN connects to each cell, and better sentences with the expanded graph’s multiple semantic levels.Source: §V · p. 10
Loading 2506.20583v1…