Source-linked AI summary
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Ryan Kiros, Ruslan Salakhutdinov, Richard S. Zemel
TL;DR
Image captioning requires models that connect visual content with both sentence meaning and grammatical generation. The paper unifies joint image-text embeddings with a structure-content neural language model in an encoder-decoder pipeline, achieving state-of-the-art-level ranking results and new best results with OxfordNet features while also revealing multimodal vector regularities.
Problem
Image captioning needs a unified way to rank image-sentence matches and generate relevant, grammatically correct descriptions from visual content.
Method
The pipeline learns a joint image-sentence embedding with LSTM sentence encoding and uses an SC-NLM decoder that disentangles sentence structure from content.
Results
The encoder matches state-of-the-art performance on Flickr8K and Flickr30K without object detections, while OxfordNet features produce new best results across metrics.
Takeaways & Limitations
The shared embedding space supports image and sentence ranking, caption generation from scratch, and multimodal vector arithmetic such as blue-to-red car analogies.
Takeaways & Limitations
Full caption-generation procedure details are deferred to supplementary material, and the learned multimodal space is not well suited for ranking sentences and images in the DeViSE comparison.
Abstract
from arXiv · showhide
Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.
1 Introduction
The paper frames image captioning as an encoder-decoder translation problem, combining joint image-text embeddings with a structure-content language model. This pipeline supports both image-sentence ranking and novel caption generation, while exposing multimodal regularities in the learned space.
- Pipeline: The encoder learns a joint image-sentence embedding, while the decoder uses a structure-content neural language model to generate words sequentially.The encoder ranks images and descriptions; the decoder conditions generation on distributed representations.
- Pipeline: LSTM sentence representations and convolutional image features are trained to rank matching images and descriptions without requiring explicit object detections.The approach combines multimodal embedding ideas with recurrent and convolutional networks.
- Multimodal regularities: The learned multimodal space supports qualitative linguistic arithmetic, such as shifting an image representation from a blue car toward red-car images by replacing “blue” with “red.”The paper examines these regularities through vector-space analogies and PCA projections.
- Related generation approaches: The approach belongs to neural caption-generation methods, alongside template-based and composition-based alternatives with different trade-offs in fluency and expressiveness.Template methods can be accurate but robotic, whereas composition-based methods target broader and more human-like captions.
- Evaluation: Automated caption evaluation has been questioned, motivating image-caption retrieval ranking as a proxy for assessing generation quality.The paper situates its ranking objective within prior concerns that Bleu and Rouge may not match human judgments.
- Encoder-decoder framing: Image captioning is treated as translation from visual input to natural-language descriptions, drawing on encoder-decoder methods from machine translation.The framing connects image-caption scoring with language-image alignments that can support generation.
2 An encoder-decoder model for ranking and generation
The paper combines an LSTM-based multimodal image-sentence encoder with a structure-content neural language model decoder. The encoder supports image-description ranking, while the decoder generates descriptions conditioned on shared multimodal representations.
- Encoder: LSTM encodes sentences into a joint image-text space, where image features are projected and image-description compatibility is learned with a ranking objective.The image representation comes from convolutional-network features, while the sentence representation is the LSTM’s final hidden state.
- Decoder: A multiplicative neural language model predicts words from previous context and a multimodal conditioning vector.The conditioning vector modulates word representations through factored tensor parameters and factor-wise interactions.
- Model choice: The multiplicative language model significantly outperforms its additive counterpart on large caption datasets, motivating the SC-NLM’s derivation from the multiplicative variant.The comparison is reported on the SBU Captioned Photo dataset.
- Decoder: The structure-content neural language model replaces the multiplicative model’s attribute vector with an additive combination of content and sentence-structure information.Structure variables are represented with learned embeddings and context matrices; the model predicts each word from prior words, forward structure context, and content.
- Training and conditioning: The SC-NLM is trained on descriptions using description embeddings, allowing text-only training and later conditioning on image embeddings from the shared space.This design permits additional monolingual text during language-model training and image-conditioned generation when no description is available.
3 Experiments
Experiments evaluate the encoder-decoder approach for image-sentence ranking, caption generation, and multimodal vector-space regularities. The model performs comparably to m-RNN, benefits substantially from OxfordNet features, and reveals a trade-off between LSTM retrieval performance and linear-encoder analogies.
- Experimental setup: The experiments use Flickr8K and Flickr30K, with 8,000 and 30,000 images respectively, each annotated by five independent sentences.Image features come from 4096-dimensional Toronto ConvNet 5 or 19-layer OxfordNet representations.
- Image-sentence ranking: The model's ranking performance is comparable to m-RNN, outperforming or matching existing methods on some metrics while trailing on others.Table 2 reports Recall@K and median rank, where higher Recall@K and lower median rank are better.
- Image-sentence ranking: Explicit embedding spaces enable retrieval with a single matrix multiply, giving them a speed advantage and better scalability than perplexity-based methods.This contrasts with m-RNN, which does not learn an explicit image-sentence embedding and uses perplexity for retrieval.
- Image-sentence ranking: Both the proposed method and m-RNN outperform existing models that integrate object detections, while OxfordNet features produce new state-of-the-art results across all metrics.The authors attribute the ranking effectiveness to LSTM cells encoding dependencies across descriptions and learning distributed sentence representations.
- Multimodal linguistic regularities: Linear encoders support multimodal analogies such as image of a blue car minus blue plus red being near images of red cars.The construction follows approximate additive relations between word and image embeddings, but these regularities are not well observed with LSTM encoders.
- Image caption generation: The authors generated descriptions for roughly 800 SBU images using an LSTM encoder and structure-content neural language model trained on Flickr30K plus Microsoft COCO.The combined training data contains over 100,000 images and over 500,000 descriptions, and nearest-neighbor captions were shown alongside generated samples.
4 Discussion
The authors identify attention-based modeling as a next step because only a small image region is often relevant during description generation. They also plan to vary decoder and encoder architectures.
- Future work: An attention-based model is being developed to align caption parts with image regions and dynamically modify decoder-conditioning vectors.The proposed attention mechanism is intended to determine where to attend next during generation.
- Future work: Planned architectural extensions include LSTM decoders and deep, bidirectional LSTM encoders.
5 Supplementary material: Additional experimentation and details
The supplementary experiments demonstrate multimodal vector-space arithmetic and describe the caption-generation procedure. Generated captions are conditioned on image-space representations and ranked by both image relevance and language fluency.
- Multimodal linguistic regularities: Figure 4 uses online query images and retrieves images from the SBU training dataset.
- Multimodal linguistic regularities: Linear multimodal encoders show regularities in cars, colors, weather, and temperature, but sensible arithmetic requires recognized image content, relevant word operations, and a corresponding image.
- Caption generation: Image-conditioned caption generation uses the embedded image or a mean embedding of its top-5 nearest words and sentences.The nearest words and sentences are selected using cosine similarity.
- Caption generation: Captions are generated by sampling a conditioning vector and a 4-to-12-word part-of-speech sequence before computing a MAP estimate.The system generates 1000 candidate descriptions per image.
- Caption ranking: Candidate descriptions are scored using cosine-based image-description similarity, a frequent-non-stopword penalty, and a Kneser-Ney trigram language model.The translation and language-model scores are combined as a weighted sum, with weights tuned manually from qualitative results.