Source-linked AI summary
Semantic Compositional Networks for Visual Captioning
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, Li Deng
TL;DR
Visual captioning needs to combine visual content with high-level semantic concepts more effectively than prior initialization- or attention-based integrations. SCN composes tag-dependent LSTM parameters using tag probabilities, achieves superior results across image and video benchmarks, and incurs added model complexity and training/testing time.
Problem
Prior methods integrate detected semantic concepts into LSTM caption generation mainly through initialization or soft attention, limiting their integration into the generation process.
Method
SCN extends each LSTM weight matrix into tag-dependent matrices weighted by inferred tag probabilities and factorizes the resulting tensor into a three-way matrix product.
Results
SCN outperforms prior state-of-the-art approaches across multiple metrics on COCO, Flickr30k, and Youtube2Text, including a 3.1-point COCO BLEU-4 improvement.
Takeaways & Limitations
SCN provides a semantic-composition framework applicable to both image and video captioning, with analyses showing captions adjust when detected tags are modified.
Takeaways & Limitations
The model has increased complexity and training/testing time, and Youtube2Text training requires additional COCO data because the video dataset is too small for reliable semantic-concept detection.
Abstract
from arXiv · showhide
A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network. The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag. In addition to captioning images, we also extend the SCN to generate captions for video clips. We qualitatively analyze semantic composition in SCNs, and quantitatively evaluate the algorithm on three benchmark datasets: COCO, Flickr30k, and Youtube2Text. Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.
1. Introduction
Visual captioning commonly uses CNN-encoded visual features and an LSTM decoder, but prior tag-based methods integrate semantic concepts only at initialization or through soft attention. SCN instead composes tag-dependent LSTM parameters, reducing factorization complexity while supporting image and video captioning.
- Visual captioning typically encodes an image or video with a CNN and generates its caption with an LSTM decoder.
- Explicit semantic tags improve captioning, but earlier methods inject them only through LSTM initialization or soft attention.
- SCN composes tag-dependent LSTM weight matrices according to image-dependent tag probabilities to represent the overall image meaning.
- Tensor factorization converts the tag-dependent weight tensor into a three-way matrix product, reducing the number of free parameters and making learning feasible.
- The paper evaluates SCN on image and video captioning, reports state-of-the-art results, and analyzes how changing tags smoothly changes generated captions.
2. Related work
Prior visual-captioning systems mainly combine CNN visual representations with recurrent language models, while another line adds semantic information. SCN differs by implicitly composing detected concepts without imposing a hierarchy among them.
- Neural captioning methods generally extract visual features with CNNs and pass them to recurrent language models for caption generation.
- Attention-based, review-module, variational-autoencoder, and novel-object methods extend visual captioning in different directions.
- Semantic-information approaches guide caption generation with retrieved sentences or detected semantic concepts, including semantic attention and joint visual-caption embeddings.
- Unlike hierarchical composition in recursive neural networks, SCN implicitly composes concepts without a hierarchical relationship among them.
- SCN uses tensor factorization on a semantic-concept vector, contrasting with related approaches based on CNN visual features or inferred scene vectors.
3. Semantic compositional networks
SCN detects semantic concepts and uses their image-dependent probabilities to compose recurrent parameters during caption generation. Visual features initialize the model, while a factorized ensemble of tag-dependent matrices conditions decoding; SCN-LSTM is evaluated on image-captioning benchmarks.
- RNN captioning: The caption probability is factorized over sequential word predictions conditioned on prior words and the visual feature.The RNN computes each conditional with a softmax over the hidden state, which is recursively updated from the previous word, state, and visual feature.
- Semantic concept detection: Semantic concepts are selected from frequent caption words and predicted for each image as a multi-label classification problem.At test time, the semantic-concept vector contains the predicted probabilities of all tags.
- Semantic composition: SCN extends each recurrent weight matrix into an ensemble of tag-dependent matrices weighted by the corresponding semantic-concept probabilities.Each tag probability is associated with slices of the input and recurrent weight tensors, so training can be viewed as jointly training K RNNs.
- Factorized SCN-RNN: Factorization makes the semantic composition tractable by sharing basis factors while using the semantic vector to modulate the composed matrices.The unfactorized parameter count is proportional to K, which is prohibitive for large tag vocabularies such as K = 1000 for COCO.
- Complexity: SCN increases model complexity and consequently increases training and testing time relative to the basic RNN.With nf = nh, the additional parameter count is 2 · nh · (nh + K).
- SCN-LSTM: The SCN-LSTM uses visual context to initialize decoding and semantic-concept-weighted LSTM parameters throughout caption generation.The paper reports SCN-LSTM rather than SCN-RNN in experiments because LSTM units are more powerful than classical RNNs.
4. Experiments
Experiments evaluate SCN-LSTM on image and video captioning benchmarks using standard features, tag detectors, beam search, and established baselines. The model achieves strong results across COCO, Flickr30k, and Youtube2Text, while qualitative analyses examine semantic composition and the role of visual features.
- Datasets: The evaluation covers COCO and Flickr30k for image captioning and Youtube2Text for video captioning, using predefined train, validation, and test splits.COCO and Flickr30k contain 123287 and 31783 images, while Youtube2Text contains 1970 video clips.
- Representations: The experiments use ResNet-152 features for images and video frames, supplemented by C3D features for videos.The image representation uses the 2048-way pool5 layer, while C3D provides a 4096-way fc7 video representation.
- Baselines and metrics: SCN-LSTM is compared with LSTM-R, LSTM-T, LSTM-RT, and LSTM-RT2 using BLEU, METEOR, ROUGE-L, and CIDEr-D.LSTM-RT2 provides the closest direct baseline because it supplies tag features at every decoding step alongside the input word.
- Image captioning results: 3.1 points: the ensemble SCN-LSTM improves the state-of-the-art COCO BLEU-4 score by 3.1 points and obtains state-of-the-art results on COCO and Flickr30k.The reported ensemble averages five SCN-LSTM models trained with different initializations.
- Image captioning results: SCN-LSTM achieves the best performance across all evaluation metrics on both c5 and c40 blind COCO test sets.The comparison is against published state-of-the-art image captioning models reported by the COCO test server.
- Video captioning results: SCN-LSTM significantly outperforms competing methods on all Youtube2Text metrics, with especially strong gains in CIDEr-D.The model also improves substantially over the LSTM-CRT2 self-comparison, and an overarching ensemble further enhances performance.
- Qualitative analysis: Qualitative examples show that changing detected tags can smoothly alter captions, while combining tags supports syntactically correct and more detailed descriptions.Examples include replacing “grass” with “bed,” switching “baby” to “girl,” and adding “baby” to produce “a baby brushing its teeth.”
- Qualitative analysis: Visual features complement tags: SCN-LSTM correctly generates “a teddy bear” where tag-only SCN-LSTM-T generates “a stuffed animal.”The comparison tests SCN-LSTM with visual feature inputs against a variant using only tag inputs.
5. Conclusion
The paper presents SCN as a framework that composes tag meanings for visual captioning and reports validation across three datasets.
- SCN composes the individual semantic meanings of tags for visual captioning.
- Each conventional LSTM weight matrix becomes a three-way matrix product with one matrix dependent on inferred tags.
- The model acts as an ensemble of tag-dependent LSTM bases, weighted according to the likelihood that each tag is present.
- Experiments on three visual captioning datasets validate the superiority of the proposed approach.
A. More results on image captioning
Figure 7 presents COCO examples comparing captions generated by two LSTM baselines with the proposed SCN-LSTM.
- Figure 7 reports detected tags and sentence-generation results on COCO.
- The first caption set is generated by LSTM-R.
- The second caption set is generated by LSTM-RT2.
- The third caption set is generated by the proposed SCN-LSTM.
B. More results on video captioning
Figure 8 presents Youtube2Text examples comparing captions from two LSTM baselines with the proposed SCN-LSTM.
- Figure 8 reports detected tags and sentence-generation results on Youtube2Text.
- The first caption set is generated by LSTM-CR.
- The second caption set is generated by LSTM-CRT2.
- The third caption set is generated by the proposed SCN-LSTM.
C. More results for Figure 4
Figure 9 compares SCN-LSTM with a variant that removes visual feature inputs and uses only tag inputs on COCO.
- Figure 9 reports detected tags and sentence-generation results on COCO.
- The first caption set is generated by SCN-LSTM.
- The second caption set is generated by SCN-LSTM-T.
- SCN-LSTM-T omits visual feature inputs and uses only tag inputs.