Source-linked AI summary
Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Lluis Castrejon, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
TL;DR
Cross-modal transfer is difficult because learned representations may not align across modalities. The paper introduces CMPlaces and regularizes convolutional networks so shared representations can emerge from scene-category supervision alone. Retrieval and visualization experiments suggest improved semantic alignment and modality-consistent units, although feature reconstructions are blurry.
Problem
Cross-modal transfer is difficult because representations learned from different modalities are not aligned, while the target object correspondences are not annotated.
Method
The paper introduces CMPlaces and two regularization approaches that align intermediate representations across five modalities using only scene-category annotations.
Results
Retrieval experiments show the proposed methods improve over baselines, while visualizations suggest units detect the same concepts across modalities without aligned training data.
Takeaways & Limitations
Weak scene-category supervision can produce shared representations whose units respond to consistent concepts across modalities and support cross-modal retrieval.
Takeaways & Limitations
Feature inversions produce blurry output images because of data compression in pool5, even within a single modality.
Abstract
from arXiv · showhide
People can recognize scenes across many different modalities beyond natural images. In this paper, we investigate how to learn cross-modal scene representations that transfer across modalities. To study this problem, we introduce a new cross-modal scene dataset. While convolutional neural networks can categorize cross-modal scenes well, they also learn an intermediate representation not aligned across modalities, which is undesirable for cross-modal transfer applications. We present methods to regularize cross-modal convolutional neural networks so that they have a shared representation that is agnostic of the modality. Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval. Moreover, our visualizations suggest that units emerge in the shared representation that tend to activate on consistent concepts independently of the modality.
MIT CSAIL
The paper marks its authors as making equal contributions.
- An asterisk denotes equal contribution.
1. Introduction
The paper studies how to learn scene representations that align concepts across very different modalities using only weak scene-category supervision. It introduces a cross-modal dataset and methods intended to make intermediate representations modality-agnostic for transfer.
- People can recognize concepts across modalities and use this ability for learning across modalities and applying knowledge in the real world.
- The paper aims to connect objects such as beds and cars across modalities without specifying object correspondences.
- The authors assembled a dataset spanning hundreds of scene types in five modalities and use only scene-category annotations to learn aligned representations.
- Two regularization approaches align intermediate convolutional representations across modalities despite supervision limited to scene categories.
- The approach builds on domain adaptation and multimodal learning while addressing substantially different modalities with minimal supervision.
- Cross-modal representations may support retrieval with queries from easier-to-produce modalities and training when target-modality data is difficult to acquire.
2. Related Work
The related work situates the paper among domain adaptation, one-shot and zero-shot learning, cross-modal retrieval, multimodal embeddings, and visual abstraction.
- Domain adaptation studies generalization from a source distribution to a different target distribution, including unsupervised and deep-network settings.
- One-shot and zero-shot learning reuse limited examples or additional textual and attribute knowledge to learn classifiers in new domains.
- Cross-modal retrieval ranks collection items by similarity to queries expressed in another modality, such as text, clip art, or sketches.
- Visual-abstraction work collects concepts in abstract domains and transfers learned dynamics or phrases to natural images.
3. Cross-Modal Places Dataset
CMPlaces is a five-modality scene dataset organized around 205 Places categories, with unpaired examples labeled by scene type. Its modalities include natural images, line drawings, cartoons, text descriptions, and spatial text images.
- CMPlaces covers natural images, line drawings, cartoons, text descriptions, and spatial text images across 205 scene categories.
- Examples are labeled with scene categories but are not paired between modalities, supporting learning strong alignments from weakly aligned data.
- The selected modalities differ substantially from natural images and are easier to generate, supporting transfer and retrieval applications.
- Line drawings were collected through Mechanical Turk and include 6,644 training examples and 2,050 validation examples.
- Descriptions were collected through Mechanical Turk, split into 4,307 training and 2,050 validation descriptions, and designed to avoid trivial category-revealing words.
- Spatial Text combines images with words positioned according to object locations and was automatically constructed from SUN images and annotations.
4. Cross-Modal Scene Representation
The paper builds cross-modal scene networks with modality-specific front ends and shared higher layers, then uses modality tuning and activation-statistics regularization to obtain modality-aligned representations from weakly aligned labels.
- Architecture: The model uses separate networks for each modality while sharing higher-level layers intended to capture modality-independent concepts.Early layers specialize in modality-specific features, whereas higher layers represent concepts such as objects.
- Architecture: Pixel modalities use a CNN to produce pool5, while descriptions use an MLP over skip-thought vectors to produce a same-dimensional representation.The separate text pathway is needed because descriptions cannot be directly fed into the CNN.
- Alignment methods: Joint end-to-end training alone produces modality-specific later-layer units, so the paper introduces two complementary alignment strategies.The methods modify fine-tuning across modalities and regularize hidden activations to have common statistics.
- Method A: Modality Tuning: Modality tuning first freezes shared higher layers while training each modality-specific network, then jointly trains the full network after unfreezing those layers.This curriculum transfers a high-level representation before allowing later layers to accommodate other modalities.
- Method B: Statistical Regularization: Statistical regularization adds a negative-log-likelihood cost over hidden activations to encourage similar activation distributions across modalities.The distributions are estimated parametrically from Places-CNN activations; the paper considers single Gaussians and Gaussian mixtures, with diagonal covariances for fewer parameters.
- Method B: Statistical Regularization: The Gaussian-mixture variant uses a more flexible mixture distribution, whose differentiable cost can be incorporated during learning by backpropagation.The mixture weights are constrained to be nonnegative and sum to one, and diagonal covariances reduce parameter count.
5. Experimental Results
The experiments evaluate aligned cross-modal representations through retrieval, unit visualizations, and feature reconstruction. Proposed methods improve retrieval and reveal shared semantic structure across modalities, while reconstructions preserve scene concepts despite modality differences.
- Cross-Modality Retrieval: Cross-modal retrieval ranks content from other modalities by cosine distance between shared-layer features and reports mean Average Precision over 1000 sampled queries.Scene labels are used to compute Average Precision for semantically related cross-modal content.
- Cross-Modality Retrieval: Method C outperforms all other reported methods, combining description retrieval comparable to method A with method B’s superior performance for other modalities.The performance gap between the proposed methods and baselines increases as modalities become more different.
- Cross-Modality Retrieval: Both method A and B improve over all baselines, while GMM statistical regularization notably improves performance over a single Gaussian.The results suggest stronger semantic alignment in fc7, with the GMM improvement attributed to increased model complexity.
- Cross-Modality Retrieval: Proposed methods generally outperform baselines for cross-modal retrieval across CNN layers, and higher-layer features yield better results because they encode higher-level scene semantics.Within-modal retrieval also increases on average after alignment.
- Hidden Unit Visualizations: Shared pool5 units can detect the same concepts across visual images and textual descriptions without explicitly aligned training data.Visualizations use maximally activating images and words from maximally activating paragraphs across 256 filters.
- Hidden Unit Visualizations: 33% of selected units activated at least one image from every modality with the proposed method, compared with 25% for the baseline.Among units with at least four selected real-modality images, the proposed method also selected at least two images per modality 19% of the time versus 14% for the baseline.
- Feature Reconstructions: Feature reconstruction learns modality-specific mappings from shared pool5 features to downsampled images, testing which concepts and statistics survive cross-modal transfer.The method trains one inverting network for each modality using shared-representation features.
- Feature Reconstructions: Reconstructions reproduce output-modality statistics and usually depict concepts from the original scene, although differing activation statistics and information loss make the task difficult.Clipart and spatial-text modalities contain less information than natural images, and pool5 compression produces blurry outputs.