Source-linked AI summary
Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network
Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, Wei Liu
TL;DR
Video captioning must represent diverse video semantics while controlling sentence syntax, challenges that prior single-feature, concatenation-based, and unguided methods incompletely address. The paper introduces cross-gated fusion with a POS sequence generator whose global syntactic guidance is adaptively incorporated into decoding. Results on MSR-VTT and MSVD show competitive performance, while POS control improves caption syntactic diversity.
Problem
Video captioning must exploit diverse video representations and syntactic structure, whereas prior methods often overlook feature relationships and sentence POS information.
Method
The model uses cross-gated fusion to combine video representations, predicts a global POS sequence, and dynamically incorporates POS guidance into word decoding.
Results
The model achieves competitive performance on MSR-VTT and MSVD, while generated global POS information controls caption syntax and improves diversity.
Takeaways & Limitations
Global POS guidance can provide syntactic control while complementing fused video representations for video caption generation.
Abstract
from arXiv · showhide
In this paper, we propose to guide the video caption generation with Part-of-Speech (POS) information, based on a gated fusion of multiple representations of input videos. We construct a novel gated fusion network, with one particularly designed cross-gating (CG) block, to effectively encode and fuse different types of representations, e.g., the motion and content features of an input video. One POS sequence generator relies on this fused representation to predict the global syntactic structure, which is thereafter leveraged to guide the video captioning generation and control the syntax of the generated sentence. Specifically, a gating strategy is proposed to dynamically and adaptively incorporate the global syntactic POS information into the decoder for generating each word. Experimental results on two benchmark datasets, namely MSR-VTT and MSVD, demonstrate that the proposed model can well exploit complementary information from multiple representations, resulting in improved performances. Moreover, the generated global POS information can well capture the global syntactic structure of the sentence, and thus be exploited to control the syntactic structure of the description. Such POS information not only boosts the video captioning performance but also improves the diversity of the generated captions. Our code is at: https://github.com/vsislab/Controllable_XGating.
1. Introduction
Video captioning must integrate diverse video semantics and syntactic structure, but prior methods often use single features, simple concatenation, or omit POS guidance. The proposed model addresses these gaps with gated fusion and global POS-guided decoding.
- Video captioning is challenging because videos contain spatial, temporal, content, motion, and speech information that single-feature approaches cannot comprehensively exploit.
- Existing multi-representation methods commonly concatenate features while neglecting relationships that help characterize video semantics.
- Prior captioning methods neglect sentence syntax, although POS sequences can guide word generation, reduce the target-word search space, and support syntactic diversity.
- The model uses a gated fusion network with cross gating to combine multiple features, predicts global POS information, and adaptively incorporates it into the decoder.
2. Related Work
Related work progresses from template-based and sequence-learning captioning toward richer video representations, but feature fusion and POS-guided generation remain limited. Existing approaches typically concatenate diverse features or rely on restricted or impractical POS information.
- Template-based methods align subjects, verbs, and objects with video content but constrain descriptions to fixed syntactic structures.
- Sequence-learning methods enable more flexible syntax and increasingly incorporate temporal, motion, semantic, and audio information.
- Existing methods generally concatenate diverse features, overlooking relationships among representations that could improve semantic characterization.
- Prior POS-based image-captioning approaches restrict POS diversity or depend on ground-truth tags that are difficult to obtain in practice.
3. Architecture
The architecture combines gated fusion of multiple video representations, POS-sequence prediction, and a decoder that uses global POS information to guide caption generation.
- Overall architecture: The model uses an encoder-decoder architecture with a gated fusion network, POS sequence generator, and description generator.The gated fusion network learns relationships among video features; the POS generator predicts global POS information, and the description generator produces captions.
- Gated fusion network: Multiple CNN-derived representations are temporally encoded before cross gating fuses their content and motion information.The cross-gating stage is designed to exploit relationships between diverse representations rather than simply concatenating them.
- POS sequence generator: The POS generator predicts a POS sequence from fused frame features using recurrent encoding and soft attention.Attention weights select fused representations relevant to the POS tag predicted at each step, while the final hidden state captures global POS-sequence information.
- Description generator: The description generator uses the predicted global POS representation and fused video features to generate the caption through a hierarchical decoder.Its two-layer LSTM combines word and POS inputs with soft attention over fused frame features.
- Description generator: At each decoding step, cross gating adaptively strengthens POS information associated with the previously generated word.The gated POS feature is then used within the decoder alongside attended fused video features.
- Training: Training proceeds in two stages: supervised POS-generator training first, followed by joint optimization of the video encoder and description generator.The first stage freezes the description generator; after POS convergence, the remaining components are trained using cross-entropy objectives.
4. Experiments
Experiments on MSR-VTT and MSVD show that gated fusion and global POS guidance improve video captioning, while POS control also changes sentence syntax and supports more specific descriptions.
- Datasets and evaluation: The model is evaluated on MSR-VTT and MSVD using BLEU@N, METEOR, ROUGE-L, and CIDEr.MSR-VTT contains 10,000 clips with 20 descriptions each, while MSVD contains 1,970 clips with roughly 40 descriptions each.
- Performance comparisons: 62.1 ROUGE-L and 53.4 CIDEr are achieved by Ours RL(IR+M), which obtains state-of-the-art performance on both metrics.Self-critical training improves all metrics except BLEU@4 compared with cross-entropy training.
- Performance comparisons: On MSVD, Ours RL(IR+M) achieves new state-of-the-art results on BLEU@4, METEOR, ROUGE-L, and CIDEr.The cross-entropy version also obtains superior scores, especially on CIDEr.
- Ablation studies: Gated fusion significantly improves all evaluation metrics over simple feature concatenation for both IR+M and I3D+M inputs.The results indicate that cross-modal semantic relationships are not captured adequately by concatenation alone.
- Ablation studies: The cross-gating mechanism enhances relevant modality information while retaining unique information through a residual structure.This distinguishes it from compact bilinear pooling, low-rank bilinear pooling, and element-wise addition.
- POS guidance and controllability: Adding global POS guidance produces the highest scores with the same IR+M and I3D+M feature pairs and enables control of overall sentence syntax.Qualitative examples show more specific captions, such as “a woman is mixing some ingredients in a bowl” and “A baseball player is hitting the ball”; modifying POS tags can add adjectives or change articles to numbers.
5. Conclusions
The proposed model combines gated fusion with a POS sequence generator to guide video captioning. It achieves competitive performance on MSR-VTT and MSVD, while POS guidance supports syntactic control and caption diversity.
- The model fuses diverse video information through cross-gating and generates global syntactic structure as captioning guidance.The POS sequence generator supplies guidance for video caption generation.
- Competitive performance on MSR-VTT and MSVD indicates the model’s effectiveness across both datasets.
- Generated global POS information can control the syntactic structure of captions and improve their diversity.
—Supplementary Material
The supplementary material adds technical details and ablation experiments on ActivityNet 1.3 for the proposed video captioning model. It covers self-critical sequence training and supplementary evaluation analysis.
- The appendix provides technical details from the submitted ICCV2019 manuscript and reports additional ablation experiments on ActivityNet 1.3.It introduces self-critical sequence training before analyzing ActivityNet 1.3 results.
- The supplementary experiments extend evaluation beyond the manuscript’s primary experiments to examine the model on ActivityNet 1.3.
1. Reinforcement Learning
The supplementary method trains the captioning model with reinforcement learning because CIDEr is discrete and non-differentiable. Self-critical sequence training uses the model’s greedy-search reward as a baseline to reduce gradient variance.
- The model uses reinforcement learning to optimize CIDEr directly because the metric is discrete and non-differentiable.Traditional optimization with cross-entropy is therefore unsuitable for directly training against this metric.
- Videos and words form the environment, while the captioning model acts as an agent whose policy predicts words and updates its hidden state.
- A baseline reward reduces the high variance caused by estimating the policy gradient from a single sampled sentence.The baseline does not change the expected gradient when it is independent of the sampled sentence.
- Self-critical sequence training uses the reward from the current model’s greedy-search sentence as the baseline reward.This avoids learning a separate reward network for baseline estimation.
- Higher-reward sampled policies are encouraged, while lower-reward strategies are suppressed during training.The self-critical procedure requires only a forward pass to estimate the baseline.
2. Experiments on ActivityNet 1.3
ActivityNet 1.3 experiments evaluate the proposed model and its components using video-captioning metrics. Cross-gating improves over feature concatenation, and adding global POS guidance yields further improvements across feature combinations.
- ActivityNet 1.3 contains 20,000 untrimmed videos with annotated events and captions, divided into 10,024 training, 4,926 validation, and 5,044 testing videos.
- Ablation Studies: EncDec+CG (IR+M) outperforms EncDec+F (IR+M) on all evaluation metrics.This comparison tests gated fusion against simple feature concatenation using IR and motion features.
- Ablation Studies: 4.3% higher CIDEr is reported for EncDec+CG (IR+M) than EncDec+F (IR+M) on ActivityNet 1.3.
- Experimental Setup: The evaluation compares BLEU@4, METEOR, ROUGE-L, and CIDEr scores on the ActivityNet 1.3 validation split.IR denotes Inception ResNet V2 content features, I3D denotes visual content features, and M denotes optical-flow motion features.
- Ablation Studies: Adding global POS information to EncDec+CG produces further improvement, with similar gains reported for I3D and motion inputs.The authors describe the gated-fusion and POS-guided approach as effective and generalizable across these feature combinations.