Source-linked AI summary
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, James Glass
TL;DR
The paper asks whether visual objects and spoken words can be jointly discovered from raw, unaligned images and speech without conventional annotations. It trains audio-visual models for whole-image and whole-caption retrieval, then examines the localized correspondences that emerge internally. The models support semantic localization, clustering, and concept discovery across the Places and ADE20k settings.
Problem
The central question is whether machines can discover visual objects and spoken words jointly from unsegmented, unaligned, and unannotated raw inputs.
Method
The paper uses spatially and temporally distributed two-branch matchmap networks trained on paired images and spoken captions with a ranking-based retrieval objective.
Results
The models learn semantic correspondences between speech frames and image regions, enabling object and word localization, audio-visual clustering, and concept discovery as by-products of retrieval training.
Takeaways & Limitations
Image-word pairs and even image-audio dictionaries can be discovered without previously learned concepts or written representations.
Abstract
from arXiv · showhide
In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer to. We demonstrate that these audio-visual associative localizations emerge from network-internal representations learned as a by-product of training to perform an image-audio retrieval task. Our models operate directly on the image pixels and speech waveform, and do not rely on any conventional supervision in the form of labels, segmentations, or alignments between the modalities during training. We perform analysis using the Places 205 and ADE20k datasets demonstrating that our models implicitly learn semantically-coupled object and word detectors.
1 Introduction
The paper asks whether machines can jointly discover visual objects and spoken words from raw, unaligned, unannotated images and speech. It addresses speech segmentation and visual object discovery together without conventional modality-specific supervision.
- 1 Introduction: The models operate on paired images and speech waveforms, with speech serving as the linguistic input rather than written text.Figure 1 presents the raw paired inputs.
- 1 Introduction: Raw speech is continuous, noisy, unsegmented, and variable, unlike preprocessed written text with separated words.The cited variability includes differences in voice, speaking rate, emotion, noise, accent, and pronunciation.
- 1 Introduction: The paper jointly treats speech segmentation and clustering as analogous to visual object discovery in images.Its goal is to address both discovery problems together.
- 1 Introduction: Speech-based vision-language learning is harder than ambient-sound learning because object-referencing speech portions are shorter and category counts are larger.Prior vision-speech work related complete speech signals and images through a global embedding, focusing on retrieval.
- 1 Introduction: The proposed model highlights image regions relevant to speech as the speech describes an image.Training uses paired images and spoken captions while leaving both modalities unsegmented, unaligned, and unannotated.
2 Prior Work
Prior work spans unsupervised visual object discovery, unsupervised speech processing, and multimodal learning, but this paper combines raw speech with visual object discovery.
- Prior Work: Unsupervised visual object discovery seeks object categories without a labeled training dataset, using approaches including clustering, topic-model ideas, and CNNs.Some CNN methods associate image patches that commonly appear adjacent to one another.
- Prior Work: Unsupervised speech processing commonly segments utterances at phone or word boundaries and clusters segments representing the same underlying unit.Popular approaches include dynamic time warping and Bayesian generative models.
- Prior Work: Vision-language research has covered captioning, visual question answering, multimodal dialogue, and text-to-image generation, mostly using written language.A growing number of studies instead learn directly from speech signals.
- Prior Work: Recent vision-sound models discover objects and ambient sounds from acoustic signals, whereas this work focuses on speech and word discovery.Combining speech with ambient sounds is identified as future research beyond the paper’s scope.
3 Spoken Captions Dataset
The study trains on a large Places image-caption collection and adds ADE20k captions with labels for fine-grained localization analysis.
- Spoken Captions Dataset: The augmented Places Audio Caption dataset contains 402,385 image/caption pairs for training and 1,000 held-out pairs for validation.The collection combines approximately 200,000 original recordings with 200,000 additional captions.
- Spoken Captions Dataset: Additional captions were collected for 9,895 ADE20k images whose scene categories also occur in Places 205.ADE20k provides pixel-level object labels for detailed localization analysis.
- Spoken Captions Dataset: ADE20k object labels combined with acoustic frame-level ASR hypotheses identify correspondences between spoken words and visual objects.This enables fine-grained evaluation of localization abilities.
4 Models
The models learn spatially and temporally distributed image and audio representations in a shared embedding space, using ranking-based image-audio retrieval to expose localized correspondences. Matchmap-based similarity functions compare image regions with audio frames, supporting later analysis of object-word localization.
- Architecture: CNN branches independently encode images and spoken captions into a shared embedding space, with distributed representations across image locations and audio time.Unlike prior global embeddings, these feature maps support direct co-localization within both modalities.
- Training objective: The ranking loss makes matched image-caption pairs more similar than mismatched pairs sampled from the same minibatch.The margin hyperparameter η is fixed to 1 in the experiments, and the similarity function is varied.
- Model use: The training criterion enables image-caption retrieval, while the paper examines object and word localization as a by-product of that training.The next analysis investigates semantic correspondences between image objects and spoken words.
- Image branch: The image branch uses a VGG16-based architecture trained end-to-end without ImageNet pre-training or conventional annotations.This contrasts with prior uses of pre-trained VGG weights and global image embeddings.
- Audio branch: The audio branch converts speech into 40-band log Mel spectrograms and outputs a feature map across the audio rather than a single embedding vector.The spectrogram uses a 25 ms Hamming window and 10 ms shift before CNN processing.
- Matchmap similarities: The matchmap tensor records localized similarity between image regions and audio segments, while alternative pooling functions emphasize different region-frame correspondences.SIMA preserves summation over image regions but selects the highest-similarity audio frame for each region.
5 Experiments
The experiments evaluate matchmap models on image-caption retrieval, speech-prompted localization, audio-visual clustering, and concept discovery using Places and ADE20k data.
- Image and Caption Retrieval: MISA achieved the best image/caption retrieval performance among the three matchmap similarity measures, with both random and ImageNet-pretrained image branches evaluated.The retrieval validation used 1,000 held-out image/caption pairs and sampled margin ranking training.
- Speech-Prompted Object Localization: Object localization was evaluated on ADE20k by converting speech-word matchmap activations into heatmaps and measuring intersection over union against referenced object masks.Scores were averaged over 100 manually selected, sufficiently frequent and unambiguous word-object pairings.
- Clustering of Audio-Visual Patterns: The clustering experiment smoothed and thresholded ADE20k matchmaps, rescaled image and speech representations, and clustered their concatenated vectors into 135 final clusters.The top 50 clusters had an average size of 44, and clusters were labeled by frequent words and object classes.
- Clustering of Audio-Visual Patterns: Clusters showed that the audio and image networks could agree on shared representations, with word and object labels jointly organized into common concepts.The networks’ agreement follows from their representations being combined through a dot product.
- Concept discovery: building an image-word dictionary: The model independently paired highly activated image concepts with highly activated spoken words, producing an image-word dictionary without using the final cross-modal activation to select images.A concept value measured agreement between the two networks from 0 to 1; values above 0.6 were found anecdotally to indicate learned concepts.
- Concept discovery: building an image-word dictionary: Across three losses, the four networks preserved the same ordering in the number of learned concepts, indicating that architecture influenced concept counts.The comparison was summarized in Figure 7.
6 Conclusions
The paper presents audio-visual matchmap networks that learn semantic correspondences between speech frames and image pixels without annotated training data. It applies these networks across retrieval, localization, clustering, concept discovery, and speech-driven highlighting, while expanding audio-caption datasets for Places and ADE20k.
- Audio-visual matchmap networks learn semantic correspondences between speech frames and image pixels without annotated training data in either modality.
- The networks support semantic image/spoken-caption search, speech-prompted object localization, audio-visual clustering, concept discovery, and real-time speech-driven semantic highlighting.
- The paper extends the Places audio caption dataset by doubling its total captions and introduces nearly 10,000 captions for ADE20k.
- Future directions include videos, environmental sounds, additional languages, richer relational datasets, and dialog feedback for language learning.