PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical LSTMs with Adaptive Attention for Visual Captioning

Jingkuan Song, Xiangpeng Li, Lianli Gao, Heng Tao Shen

arXiv:1812.11004v1cs.CV

TL;DR

Visual captioning models often apply attention to every word even though non-visual words can rely on language context, while single-layer LSTMs limit hierarchical representation. The paper proposes hLSTMat, which combines spatial or temporal attention, adaptive selection between visual and language information, and hierarchical LSTMs for image and video captioning. Experiments report state-of-the-art performance on both tasks for most evaluation metrics, with further gains from spatial-temporal networks on video captioning benchmarks.

  • Problem

    Existing visual captioning models apply attention to visual and non-visual words alike, and most use single-layer LSTMs rather than simultaneously modeling visual information and language context.

  • Method

    hLSTMat combines an encoder, hierarchical LSTMs, spatial or temporal attention, and adaptive attention that selects between visual information and language context.

  • Results

    The approach achieves state-of-the-art performance on both MSVD and MSR-VTT, while ParA and Two-stream further improve video captioning across three datasets.

  • Takeaways & Limitations

    The framework provides a unified approach for image and video captioning that automatically decides when and where to use visual information and language context.

Abstract

from arXiv · show

Recent progress has been made in using attention based encoder-decoder framework for image and video captioning. Most existing decoders apply the attention mechanism to every generated word including both visual words (e.g., "gun" and "shooting") and non-visual words (e.g. "the", "a"). However, these non-visual words can be easily predicted using natural language model without considering visual signals or attention. Imposing attention mechanism on non-visual words could mislead and decrease the overall performance of visual captioning. Furthermore, the hierarchy of LSTMs enables more complex representation of visual data, capturing information at different scales. To address these issues, we propose a hierarchical LSTM with adaptive attention (hLSTMat) approach for image and video captioning. Specifically, the proposed framework utilizes the spatial or temporal attention for selecting specific regions or frames to predict the related words, while the adaptive attention is for deciding whether to depend on the visual information or the language context information. Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the caption generation. We initially design our hLSTMat for video captioning task. Then, we further refine it and apply it to image captioning task. To demonstrate the effectiveness of our proposed framework, we test our method on both video and image captioning tasks. Experimental results show that our approach achieves the state-of-the-art performance for most of the evaluation metrics on both tasks. The effect of important components is also well exploited in the ablation study.

1 INTRODUCTION

The introduction identifies two limitations in visual captioning: attention is applied indiscriminately across words, and most models use single-layer LSTMs. It proposes hLSTMat, combining adaptive attention with hierarchical LSTMs for video and image captioning.

  • Motivation: Video captioning is challenging because videos contain diverse objects, scenes, actions, attributes, and salient contents.The task integrates visual understanding with natural language generation and supports applications including retrieval, understanding, navigation, and subtitling.
  • Problem: Current visual attention models use visual information for every generated word, although non-visual words can rely on language context and attention may mislead.The paper distinguishes visual words such as “man” and “gun” from non-visual words such as “the” and “is.”
  • Problem: Most visual captioning tasks use a single-layer LSTM, limiting hierarchical representation of visual and linguistic information.The introduction contrasts this practice with the richer processing associated with hierarchical deep networks.
  • Approach: The proposed hLSTMat framework combines spatial or temporal attention with adaptive attention to decide where to use visual information and when to use language context.It is presented as a unified encoder-decoder framework for visual captioning.
  • Approach: Hierarchical LSTMs combine low-level visual information with high-level language context and can also support sequential deliberation to refine an initial representation.The framework is evaluated on both video and image captioning tasks.

2 RELATED WORK

Related work progresses from basic encoder-decoder captioning to visual, semantic, and reinforcement-learning-based attention. These approaches improve feature selection or optimization, while hLSTMat is positioned within this progression as a model innovation.

  • Overview: Image and video captioning translate visual content into natural-language sentences describing image or video content.Prior work mainly focuses on feature extraction and model innovation.
  • Basic Encoder-Decoder: Basic encoder-decoder methods use CNN representations and recurrent architectures such as LSTMs to generate captions.Early video-captioning systems directly connect visual convolutional models with deep LSTM networks.
  • Visual Attention: Visual attention methods weight image regions or video frames to guide where captioning models look during sentence generation.Video methods also incorporate local dynamics and global temporal structure or combine temporal, motion, and semantic attention.
  • Semantic Attention: Semantic attention incorporates attributes or concepts generated by pretrained models to emphasize semantically important concepts or regions.These methods weight the relative strength of attention across multiple concepts.
  • Deep Reinforcement Learning: Reinforcement-learning approaches optimize captioning with policy-gradient or policy-and-value-network strategies using rewards such as CIDEr or SPICE.These methods treat sentence generation and evaluation as components of a reinforcement-learning process.

3 HIERARCHICAL LSTM WITH ADAPTIVE ATTENTION FOR VISUAL CAPTIONING

The method uses an encoder, a hierarchical LSTM decoder with adaptive attention, and caption losses. It extends basic LSTM decoding by combining visual features and language context while learning to generate descriptions.

  • Basic LSTM: An LSTM uses input, forget, and output gates plus a memory cell to update its internal state over time.The gates regulate whether information is accepted, forgotten, or exposed, while the formulation is parameterized by learned matrices and biases.
  • Basic Visual Captioning: The visual captioning pipeline encodes an image or video representation and decodes it into a word sequence with an LSTM-based decoder.The decoder updates recursively until an end-of-sentence tag is generated.
  • hLSTMat Framework: hLSTMat consists of an encoder, an attention-based hierarchical LSTM decoder, and loss functions.The encoder is intended to produce compact, representative visual features for decoding.
  • Hierarchical Decoder: The hierarchical decoder uses multiple LSTM layers as a processing pipeline to enrich representation beyond the single-layer vanilla decoder.Its organization can process different levels of information before generating the next word.
  • Adaptive Attention: Adaptive attention decides whether the decoder should depend on visual information or language context information.This mechanism is defined on the hierarchical LSTM and controls the source of information used during caption generation.
  • Losses: The maximum-likelihood loss optimizes model parameters by maximizing the probability of each ground-truth word conditioned on previous words and input features.The loss is defined over the caption length, ground-truth prefix, input representation, and model parameters.

4 HLSTMAT FOR VIDEO CAPTIONING

The video-captioning hLSTMat decoder combines hierarchical LSTMs with temporal and adaptive attention to select relevant frames and determine when to use visual evidence versus language context. It also supports appearance-motion fusion architectures for richer video representations.

  • Architecture: The video-captioning system contains a CNN encoder, an attention-based hierarchical LSTM decoder, and loss components.The decoder integrates two LSTM layers and attention mechanisms before predicting vocabulary distributions.
  • Hierarchical decoder: The bottom LSTM decodes visual features, while the top LSTM mines deeper language-context information for caption generation.The top layer receives the bottom layer output together with its previous hidden and memory states.
  • Temporal attention: Temporal attention dynamically weights video frames instead of averaging them, using the current bottom-layer hidden state to identify salient features.Attention scores are normalized with softmax, and the resulting weights quantify feature relevance.
  • Adaptive attention: Adaptive temporal attention determines whether the decoder should use visual information or sentence context, using little visual information for non-visual words.A sigmoid-based filtering step further regulates the visual information passed to the decoder.
  • Multiple features: Because appearance alone cannot distinguish some motion-dependent actions, the method evaluates concatenation fusion, two-stream, and parallel adaptive temporal attention architectures.The parallel model combines appearance and motion evidence while avoiding the doubled training time and parameter count of two-stream networks.

5 EXPERIMENTS FOR VIDEO CAPTIONING

The video-captioning experiments evaluate hLSTMat across three established datasets using standard captioning metrics and specified preprocessing, optimization, and decoding procedures.

  • Datasets: The evaluation compares hLSTMat on MSVD, MSR-VTT, and LSMDC, three publicly available video-captioning datasets.The datasets differ in scale and content, including short clips, web videos, and movie descriptions.
  • Datasets: MSVD contains 1,970 clips and approximately 80,000 description pairs, while MSR-VTT contains 10,000 clips with about 200,000 clip-sentence pairs.LSMDC contains 118,081 video clips and 118,114 aligned sentences from 202 movies.
  • Preprocessing and decoding: Descriptions are tokenized into dataset-specific vocabularies, and BOS and EOS tags delimit sentence generation during training and testing.Testing starts from BOS and selects the maximum-probability word until EOS.
  • Optimization: Training uses 512-unit LSTMs, 512-dimensional word embeddings, mini-batches, Adadelta optimization, dropout, and element-wise gradient clipping.The stated dropout rate is 0.5 and gradients are clipped at 10.
  • Evaluation: Performance is assessed with the standard BLEU, METEOR, and CIDEr metrics.

5.3 The Effect of Different CNN Encoders

The encoder study compares CNN feature extractors on MSVD and MSR-VTT, finding that ResNet-152 performs best while C3D performs worst among the reported alternatives.

  • Experimental setup: The encoder experiments are conducted on MSVD and MSR-VTT using multiple CNN feature extractors.The study covers C3D, GoogleNet, Inception-V3, ResNet-50, ResNet-101, and ResNet-152 in the broader encoder analysis.
  • MSVD: 82.9% B@1, 72.2% B@2, 63.0% B@3, 53.0% B@4, and 33.6% METEOR are obtained with ResNet-152 on MSVD.Inception-v3 is a close competitor with 82.7% B@1, 72.0% B@2, 62.5% B@3, 51.9% B@4, and 33.5% METEOR.
  • MSR-VTT: ResNet-152 performs best and C3D obtains the worst results in the additional MSR-VTT feature comparison.The comparison includes C3D, Inception-v3, and ResNet-152.

5.4 Architecture Exploration and Comparison

The study isolates hierarchical LSTMs, temporal attention, and adaptive attention on MSVD, finding that their combination gives the strongest overall results.

  • Component comparison: hLSTMat achieves the best results across all reported metrics: 82.9% B@1, 72.2% B@2, 63.0% B@3, 53.0% B@4, 33.6% METEOR, and 73.8% CIDEr.The comparison includes basic LSTM, adaptive-attention variants, hLSTMt, MP-LSTM, and SA.
  • Temporal attention: Temporal attention improves basic LSTM by 1% B@1, 1% B@2, 1.9% B@3, 1.7% B@4, 0.6% METEOR, and 2.1% CIDEr.This comparison is between basic LSTM and SA, which adds temporal attention.
  • Adaptive attention: Adaptive attention improves caption prediction by choosing between visual information and language context, although its gain is smaller than temporal attention.The analysis identifies adaptive attention as important but less influential than temporal attention.
  • Hierarchical structure: Hierarchical LSTMs improve over SA by 0.9% B@1, 1.6% B@2, 0.4% B@3, 0.8% B@4, 0% METEOR, and 1.5% CIDEr.The comparison attributes these gains to adding the hierarchical structure.

5.5 The Effect of Spatial-temporal hLSTMat

The experiments examine multiple-feature temporal and spatial-temporal hLSTMat variants on MSVD, showing that Two-stream generally performs best while costing more to train.

  • Architecture variants: Two-stream consistently outperforms ParA and ConF across all reported evaluation metrics.The comparison evaluates combinations of appearance and motion features in multiple hLSTMat architectures.
  • Feature combinations: Combining multiple features generally performs better than a single feature for most video-captioning metrics, although a single appearance feature gives the best CIDEr.The experiments use C3D and ResNet-152 as multiple features in one group and compare temporal and spatial-temporal variants.
  • Computational cost: Two-stream doubles ParA in the number of training parameters and training time.The performance ranking is Two-stream first, ParA second, and ConF worst.
  • Architecture variants: Two-stream achieves 84.1% B@1, 74.9% B@2, 66.2% B@3, 56.2% B@4, 34.7% METEOR, and 81.1% CIDEr with Inception-v3 and ResNet-152.This is the strongest reported result in the Inception-v3 replacement experiment.

5.6 Compare with the-state-of-the-art Methods

Across MSVD, MSR-VTT, and LSMDC comparisons, hLSTMat variants achieve strong video-captioning results, with Two-stream often leading but dataset coverage affecting scores.

  • MSVD results: hLSTMat achieves 53.0% B@4 and 33.6% METEOR using static frame-level features, exceeding HRNE and HRNE-SA on both metrics.The reported HRNE values are 43.6% B@4 and 32.1% METEOR; HRNE-SA reports 43.8% B@4 and 33.1% METEOR.
  • MSVD results: Two-stream achieves 84.1% B@1, 74.9% B@2, 66.2% B@3, 56.2% B@4, 34.7% METEOR, and 81.1% CIDEr on MSVD.It outperforms the compared two-feature approaches on the reported metrics.
  • Human evaluation: User questionnaires show higher Overall Quality and slightly higher Caption Accuracy for the proposed method, but lower Information Coverage than p-RNN.The questionnaire involved ten users with different academic backgrounds.
  • MSR-VTT results: Two-stream reaches 40.0% @B4 and 27.3% METEOR on MSR-VTT, while hLSTMat (R) reaches 38.3% @B4 and 26.3% METEOR.Two-stream improves over hLSTMat using only temporal attention by 1.7% @B4 and 1.0% METEOR.
  • LSMDC results: On LSMDC, Two-stream obtains the highest CIDEr among comparable methods at 10.4%, while CT-SAN leads B@4 and ROUGE.The passage reports CT-SAN at 0.8% B@4 and 15.9% ROUGE, with another method best on METEOR.
  • Dataset considerations: LSMDC scores are low across methods because each of its 118,081 clips has approximately one ground-truth sentence, limiting description coverage.The passage contrasts this with MSR-VTT, where each video contains approximately 20 sentences.

5.7 Feature Weights Analysis

Feature-weight analysis shows that ParA shifts reliance among language context, appearance, and motion features during caption generation according to word type.

  • Weight visualization: Fig. 6 plots ResNet152, C3D, and hidden-state weights over caption-generation time, with weights summing to 1.Words are placed at the time steps when they are generated, and the vertical axis shows weight scores.
  • Language context: Language-context weight generally increases for non-visual words such as “a”, “the”, “are”, and “is”.In one example, the hidden-state weight exceeds 0.7 for both “a” and “the”.
  • Visual features: Appearance-feature weight tends to be larger for nouns, whereas motion-feature weight tends to be higher for verbs and actions.Examples include nouns such as “man”, “cat”, “dog”, and “horse”, and verbs such as “walking”, “plays”, “barking”, and “slicing”.

6 HLSTMAT FOR IMAGE CAPTIONING

The image-captioning model combines region-level visual features, hierarchical LSTMs, residual connections, and adaptive attention to generate and refine captions. It is trained first with maximum likelihood and then fine-tuned using reinforcement learning with CIDEr and contrastive loss.

  • Architecture: The image-captioning system contains a CNN encoder, an attention-based hierarchical LSTM decoder, and loss functions.It extracts global and region-level visual features before decoding captions.
  • Architecture: The two LSTM blocks first generate a preliminary caption and then act as a proofreading process to refine it.The first block prepares hidden states and visual attention, while the second performs deliberate refinement.
  • Optimization: Residual shortcut connections provide an additional word shortcut alongside the LSTM temporal path to reduce vanishing gradients.The shortcut concatenates the relevant hidden-state information using a learned parameter.
  • Attention: Visual attention assigns weights to image regions so the model can selectively use region features when generating each word.The attended visual information is combined with recurrent states to produce the primary word.
  • Attention: Adaptive attention combines visual features with language context to determine how much information should support each predicted word.The second LSTM computes attention over image-region features and sequential context before word prediction.
  • Training: Training first minimizes maximum-likelihood loss, then fine-tunes the model with reinforcement learning using CIDEr and contrastive loss rewards.The contrastive loss uses matched and mismatched image-caption pairs with a hinge-loss objective.

7 EXPERIMENTS FOR IMAGE CAPTIONING

Experiments evaluate the deliberation-based image-captioning model on COCO and Flickr30K using standard captioning metrics and ablations. Deliberation and reinforcement learning improve performance, while the full model achieves strong single-model results against established systems.

  • Datasets and Metrics: The evaluation uses COCO and Flickr30K, with captions truncated to 16 words and standard BLEU, ROUGE, METEOR, CIDEr, and SPICE metrics.COCO also includes SPICE subclass scores for color, attribute, cardinality, object, relation, and size.
  • Ablation Study: The ablation study compares models with and without deliberation and reinforcement learning on COCO.The four configurations are DA-De-RF, DA-De+RF, DA+De-RF, and DA+De+RF.
  • Ablation Study: 2% BLEU4, 1% METEOR, 1.6% ROUGE, 7% CIDEr, and 1.1% SPICE are the gains of DA+De-RF over DA-De-RF.DA+De-RF outperforms DA-De-RF on all 12 reported metrics.
  • Qualitative Analysis: The second residual attention layer refines activities and object relationships, while the first layer detects primary objects and activities.The visual weights in the second layer are higher for object-related words such as “car” and “pizza.”
  • Qualitative Analysis: Contrastive-loss reinforcement learning improves discriminability by producing captions that distinguish similar scenes, such as a street versus a park.The qualitative comparison reports that the non-reinforcement model generates the same description for both images.
  • Benchmark Results: On Flickr30K, DA exceeds Adaptive by 6.1% BLEU-1, 5.7% BLEU-2, 4.9% BLEU-3, 4.3% BLEU-4, 2.6% METEOR, and 13.5% CIDEr.The reported improvements are especially large for BLEU-n and CIDEr.
  • Benchmark Results: On the COCO single-model test split, DA performs best on BLEU4, METEOR, ROUGE, CIDEr, and SPICE among the compared systems.Up-Down remains strongest for the Cardinality and Relation SPICE subclasses.

8 CONCLUSION AND FUTURE WORK

The paper concludes that hLSTMat combines hierarchical LSTMs with temporal and adaptive temporal attention for video captioning, deciding when to use visual information or sentence context. It reports state-of-the-art performance on MSVD and MSR-VTT, while ParA and Two-stream further improve video-captioning results across three datasets.

  • Conclusion: hLSTMat integrates hierarchical LSTMs, temporal attention, and adaptive temporal attention to select visual or sentence-context information during caption generation.The framework simultaneously considers low-level video features and language context.
  • Conclusion: hLSTMat achieves state-of-the-art performance on the MSVD and MSR-VTT video-captioning datasets.The conclusion reports this outcome for the proposed framework.
  • Future Work: ParA and Two-stream further improve video-captioning performance on all three evaluated datasets.These are proposed as spatial-temporal networks extending the framework.
Loading 1812.11004v1…