Source-linked AI summary
Deep Multimodal Semantic Embeddings for Speech and Images
David Harwath, James Glass
TL;DR
The paper addresses whether image semantics can be learned directly from spoken captions instead of text transcripts. It combines visual and speech CNNs with shared-space alignment, evaluates the model on Flickr8k image search and annotation, and identifies data and segmentation constraints for future work.
Problem
The paper investigates replacing parallel text transcripts with relevant spoken captions for learning correspondences between speech and visual objects.
Method
The model uses CNNs for image regions and word-level speech, maps both into a shared semantic space, and trains on paired Flickr8k images and spoken captions.
Results
The model was evaluated on Flickr8k image search and annotation using recall@10, with comparisons against word-vector and text-based systems.
Takeaways & Limitations
Paired images and natural-language spoken captions can support joint semantic representations that align salient visual objects with associated caption words.
Takeaways & Limitations
The current setting requires forced alignment and pre-segmented word audio, while the speech CNN remains behind systems using ground-truth text.
Abstract
from arXiv · showhide
In this paper, we present a model which takes as input a corpus of images with relevant spoken captions and finds a correspondence between the two modalities. We employ a pair of convolutional neural networks to model visual objects and speech signals at the word level, and tie the networks together with an embedding and alignment model which learns a joint semantic space over both modalities. We evaluate our model using image search and annotation tasks on the Flickr8k dataset, which we augmented by collecting a corpus of 40,000 spoken captions using Amazon Mechanical Turk.
1. INTRODUCTION AND RELATED WORK
The paper replaces text transcripts with spoken captions and learns to associate speech segments with visual objects. It extends image-text alignment work by modeling image and speech semantics directly from audio.
- Motivation: The model learns associations between spoken instances of words and corresponding objects in image scenes.It uses paired images and spoken audio captions segmented at the word level.
- Approach: A pair of CNNs represents visual objects and spoken words, while an alignment model maps them into a shared semantic embedding space.
- Related Work: Earlier multimodal research primarily modeled images with text, including object identification and alignment between textual descriptions and visual regions.
- Related Work: Caption-generation research inspired the paper’s text-to-image alignment formulation, although this work does not generate image captions.
2. MODEL DESCRIPTION
The model extracts fixed-dimensional representations of image regions and spoken words, then aligns them in a shared semantic space. It trains this correspondence with image-caption matching and word-to-region grounding.
- 2.1. Region Convolutional Neural Network: RCNN proposals produce 20 image regions per image, represented by 4096-dimensional feature vectors.The set includes the top 19 detected regions and the entire image frame.
- 2.2. Spectrogram Convolutional Neural Network: The speech CNN operates directly on pre-segmented single-word audio rather than performing word segmentation during inference.Word spectrograms are padded or truncated to a fixed-duration window for CNN processing.
- 2.2. Spectrogram Convolutional Neural Network: Word spectrograms are created from forced-aligned audio using 25-millisecond windows, 10-millisecond shifts, and 40 mel-spaced filterbanks.The spectrograms are mean-subtracted, variance-normalized, and adjusted to a fixed duration by zero-padding or truncation.
- 2.2. Spectrogram Convolutional Neural Network: The speech CNN retains 1024-dimensional activations from the fully connected layer before its softmax classification output as word representations.
- 2.3. Embedding Alignment Model: Separate image and speech transforms map region and word vectors into a shared h-dimensional semantic embedding space.The image transform is affine, while the speech transform is nonlinear with a ReLU function.
- 2.3. Embedding Alignment Model: The alignment objective scores each word by its maximum inner-product similarity with image fragments and sums these scores across the caption.A max-margin objective trains matching image-caption pairs to score higher than mismatched pairs.
3. DATA
The authors augment Flickr8k with spoken captions collected through Mechanical Turk and verify the recordings before word-level preprocessing. The resulting corpus supports supervised feature extraction and alignment experiments.
- Audio collection: The authors collect spoken recordings for all 40,000 captions through Mechanical Turk using a browser-based audio interface.Workers read displayed captions aloud, replay recordings, and can re-record when recognition feedback indicates a problem.
- Dataset: The collected utterances are divided into 30,000 training, 5,000 development, and 5,000 testing examples covering an 8,918-word vocabulary.The splits correspond to Flickr8k’s existing training, validation, and testing partitions.
- Data verification: 11.67% word error rate on the test set provides an additional indication that the collected audio is relatively high quality.The recognizer is also used to force-align audio to transcripts and segment it at the word level.
- Preprocessing: The model uses off-the-shelf RCNN visual features and supervised pretraining on Wall Street Journal speech because Flickr8k is comparatively small.The visual object embeddings are 4096-dimensional.
4. EXPERIMENTS
The experiments train and evaluate the embedding-alignment model on Flickr8k image search and annotation, while comparing speech-based features with text-based and prior systems. Additional isolated-word experiments examine CNN architecture choices.
- Training: The embedding and alignment model is trained for 20 epochs on Flickr8k’s 6,000-image training set using 30,000 captions.Training uses stochastic gradient descent with batches of 40 images, and embedding dimensions between 512 and 1024 perform well.
- Evaluation: Image search retrieves an image for a caption, whereas image annotation retrieves a matching caption for an image; both use recall@10.The evaluation uses 1,000 test images with five captions each.
- Evaluation: Table 1 compares the Spectrogram CNN with 200-dimensional word vectors and prior systems on Flickr8k image search and annotation.The text-plus-word-vector system outperforms Karpathy’s earlier system.
- Word CNN experiments: 75.5% isolated-word classification accuracy is achieved by a two-layer fully connected DNN, while adding a third layer reduces accuracy to 69.5%.The authors speculate that the training set is too small for the deeper network.
- Word CNN experiments: 84.2% accuracy is achieved by replacing the first fully connected layer with a 64-unit convolutional layer.All tested networks exceed 90% top-5 accuracy.
5. CONCLUSION
The paper presents a joint semantic representation for spoken words and visual objects, learned from paired images and spoken captions. It evaluates cross-modal retrieval and identifies segmentation, data scale, and supervised pretraining as directions for further work.
- Contributions: The model learns a joint semantic representation over spoken words and visual objects using weak labels from paired images and spoken captions.It aligns salient visual objects with associated caption words.
- Evaluation: The authors evaluate the model on Flickr8k image search and annotation and compare it with systems that access ground-truth text.The paper also illustrates inferred word-to-object alignments on Flickr8k images.
- Future work: Future work includes incorporating word-level segmentation into alignment to remove the need for forced alignment.The authors also propose increasing training data and jointly training feature extractors with the alignment model.