Source-linked AI summary
See, Hear, and Read: Deep Aligned Representations
Yusuf Aytar, Carl Vondrick, Antonio Torralba
TL;DR
The paper targets representations that align the same concept across vision, sound, and language. It learns them from synchronous image–sound and image–text data, and reports improved retrieval and classification, including sound–text transfer without sound–text training pairs.
Problem
The paper seeks representations that remain aligned when the same concept appears in different modalities, such as a pool image, splashing sound, or sentence.
Source: §1 · p. 1Method
A deep convolutional network learns shared representations across vision, sound, and language from in-the-wild data using synchronized modality pairs.
Source: §1 · p. 1Results
Experiments report improved retrieval and classification in challenging in-the-wild settings, sound–text alignment without training pairs, and modality-independent high-level concept units.
Source: §1 · pp. 1–2, §5 · p. 8Takeaways & Limitations
The authors suggest images may act as a bridge through which the representation aligns sound and text despite never training on those pairs.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language. By leveraging over a year of sound from video and millions of sentences paired with images, we jointly train a deep convolutional network for aligned representation learning. Our experiments suggest that this representation is useful for several tasks, such as cross-modal retrieval or transferring classifiers between modalities. Moreover, although our network is only trained with image+text and image+sound pairs, it can transfer between text and sound as well, a transfer the network never observed during training. Visualizations of our representation reveal many hidden units which automatically emerge to detect concepts, independent of the modality.
1. Introduction
The paper seeks representations that place the same concept near one another across vision, sound, and language, and learns them from large amounts of in-the-wild data. Its experiments report modality-independent concept units and transfer between sound and text despite training without sound–text pairs.
1. Introduction
- Modality robustness means similar representations for the same concept across modalities, such as a pool image, splashing sound, and sentence.The paper frames aligned representations as a way to connect relationships between modalities, with applications including transferring learned knowledge.Source: §1 · p. 1
- The network learns a shared representation for vision, sound, and language from in-the-wild data using synchronization and transferred visual models.The paper describes an unsupervised synchronization-based approach and a method for transferring discriminative visual models to other modalities.Source: §1 · p. 1
- The representation transfers between sound and text despite training only on image–text and image–sound pairs, and upper-layer units detect concepts across modalities.The paper reports this transfer even though the network never saw sound–text pairs during training.Source: §1 · pp. 1–2
- The paper’s primary contribution is applying standard methods at large scale to learn aligned representations across three natural modalities from synchronized data.The authors describe this large-scale, three-modality application as novel to their knowledge.Source: §1 · p. 2
1.1. Related Work
- Unlike sound–language retrieval work,,, this paper uses vision as an intermediary; images are shared across its image–sound and image–text pairs.The related-work discussion also distinguishes this three-modality alignment from prior image–language approaches.Source: §1.1 · p. 2
2. Datasets and Modalities
The datasets pair images separately with natural environmental sound or text, providing large-scale synchronous data while leaving sound–text pairs out of training.
2. Datasets and Modalities
- The authors chose sound, vision, and language because these modalities are frequently used in everyday situations.Figure 2 presents examples of the data used.Source: §2 · p. 2
Sound
- The sound data comprise over 750,000 Flickr videos and 377 days of continuous audio, with 85% of sound files used for training.The remaining 15% are used for evaluation; spectrograms are extracted from approximately five seconds of audio.Source: §2 · pp. 2–3
Language
- The language data combine COCO’s 400,000 sentences and 80,000 images with Visual Genome’s 4,200,000 descriptions and 100,000 images.Sentences are preprocessed by removing English stop words and embedding each word with word2vec.Source: §2 · p. 3
Images
- Nearly a million images come from sound and language datasets, each synchronized with sound or text, but not both.Images are channelwise mean-subtracted, and the paired datasets’ train/test splits are retained.Source: §2 · p. 3
Synchronization
- Training uses image–sound and image–text pairs but no sound–text pairs; 1,000 worker-written video descriptions are reserved to test sound–text transfer.The authors propose that images may serve as a bridge between sound and text.Source: §2 · p. 3
3. Cross-Modal Networks
The network maps image, sound, and sentence inputs into a shared representation, combining teacher-based transfer and pairwise ranking objectives with modality-specific pathways and shared upper layers.
3. Cross-Modal Networks
- The network maps paired samples from two modalities to representations that should be close under a distance metric, while avoiding the trivial constant-output solution.The method is described for two modalities and generalized to three.Source: §3 · p. 3
3.1. Alignment by Model Transfer
- A teacher model supplies class-probability targets for student models; sharing upper-layer parameters encourages internal representations to align across modalities.The transfer objective alone aligns at the teacher's predicted categories, not in the students' internal representations.Source: §3.1 · p. 3
3.2. Alignment by Ranking
- The ranking loss brings paired examples together and pushes mismatched pairs apart up to a margin, using cosine similarity as the pair score.The loss uses negative examples to make representations both aligned and discriminative.Source: §3.2 · p. 3
3.3. Learning
- Training combines model-transfer and ranking losses on different network layers; ranking supervision covers vision–text and vision–sound pairs, not sound–text pairs.The paper expects vision to bridge sound and text alignment.Source: §3.3 · p. 4
3.4. Network Architecture
- Separate sound, text, and vision pathways feed common shared layers, with each pathway adapting to modality-specific features and shared layers to modality-robust features.The sound pathway processes spectrograms with one-dimensional convolutions; text uses word2vec embeddings and a one-dimensional convolutional network; vision follows the Krizhevsky architecture.Source: §3.4 · pp. 4–5
3.5. Implementation Details
- Implementation uses 200-example batches for 50,000 iterations; sound inputs are 500 × 257 spectrograms, while text inputs are padded or cropped to 16 words.The text word vectors are 300-dimensional, and the sound and text pathways each produce 9,216-dimensional vectors for the shared layers.Source: §3.5 · pp. 5–6
- Table 3 reports classifier-transfer accuracy across training and testing modalities, suggesting better alignment than baselines without labeled examples in the target modality.The table caption says a dash marks comparisons unavailable because CCA works with only two views.Source: Table 3 · p. 6
4. Experiments
The experiments evaluate cross-modal retrieval, transfer between sound and text, classifier transfer across modalities, and hidden-unit visualizations. They report alignment across modalities, transfer to unseen modality pairs, and emerging modality-independent object detectors.
4.1. Experimental Setup
- Models are trained on training splits, tuned on validation data, and evaluated on held-out sets with 5,000 video/sound pairs and 5,000 image/text pairs.Networks are trained from random initialization; overlapping COCO and Visual Genome images are assigned to the same split.Source: §4.1 · p. 6
4.2. Cross Modal Retrieval
- Our representation significantly beats baselines overall, but only marginally beats CCA for image–text retrieval; image–sound retrieval is harder.Retrieval uses normalized last-hidden-layer features and cosine similarity.Source: §4.2 · pp. 6–7
4.3. Sound and Text Transfer
- Despite no sound/text training pairs, deep representations outperform cluster CCA and linear regression by over 100 points, and transfer is only slightly worse than sound/images.The authors suggest high-level features are easier to align across modalities.Source: §4.3 · p. 7
4.4. Zero Shot Classifier Transfer
- Up to 10% gain over CCA and Linear regression; cross-modal classification also achieves significant accuracies without target-modality examples.The authors attribute the representation’s discriminative alignment to combining paired ranking and model transfer objectives.Source: §4.4 · p. 8
4.5. Visualization
- Many shared-layer units automatically detect high-level concepts, and some appear to detect objects independently of modality.The authors identify highly activating validation inputs for units in the last hidden layer across modalities.Source: §4.5 · p. 8
5. Conclusion
The paper learns aligned representations across vision, sound, and language from large-scale in-the-wild data. Experiments report improved retrieval and classification, suggest alignment between sound and text despite no sound–text training pairs, and reveal modality-independent concept units.
5. Conclusion
- The deep convolutional network learns cross-modal representations from over a year of video and millions of sentences.Source: §5 · p. 8
- The aligned representation improves retrieval and classification performance in challenging in-the-wild situations.Source: §5 · p. 8
- Although trained only on image–text and image–sound pairs, the network empirically suggests an alignment between sound and text, possibly using images as a bridge.Source: §5 · p. 8
- Visualizations reveal high-level concept units that emerge independently of modality.Source: §5 · p. 8