Source-linked AI summary
DeMIAN: Deep Modality Invariant Adversarial Network
Kuniaki Saito, Yusuke Mukuta, Yoshitaka Ushiku, Tatsuya Harada
TL;DR
The paper addresses the costly acquisition of labeled multimodal data by learning representations shared across modalities. DeMIAN combines paired-sample matching with adversarial distribution alignment, and achieves better classification performance, including state-of-the-art results on benchmark zero-shot datasets.
Problem
Collecting paired samples with labels is costly, motivating modality-invariant representations learned from paired unlabeled modalities and separately labeled single-modal data.
Source: §1 · p. 1Method
DeMIAN uses adversarial training to align modality distributions while minimizing distances between paired representations.
Source: §1 · p. 2, §3.3 · p. 5Results
Takeaways & Limitations
The learned modality-invariant representations support shared representation learning and zero-shot learning across modalities.
Source: §1 · p. 2, §5 · p. 8The approach assumes that making modality distributions similar can produce a classifier that works well for both modalities.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Obtaining common representations from different modalities is important in that they are interchangeable with each other in a classification problem. For example, we can train a classifier on image features in the common representations and apply it to the testing of the text features in the representations. Existing multi-modal representation learning methods mainly aim to extract rich information from paired samples and train a classifier by the corresponding labels; however, collecting paired samples and their labels simultaneously involves high labor costs. Addressing paired modal samples without their labels and single modal data with their labels independently is much easier than addressing labeled multi-modal data. To obtain the common representations under such a situation, we propose to make the distributions over different modalities similar in the learned representations, namely modality-invariant representations. In particular, we propose a novel algorithm for modality-invariant representation learning, named Deep Modality Invariant Adversarial Network (DeMIAN), which utilizes the idea of Domain Adaptation (DA). Using the modality-invariant representations learned by DeMIAN, we achieved better classification accuracy than with the state-of-the-art methods, especially for some benchmark datasets of zero-shot learning.
1. Introduction
The paper targets modality-invariant representations that align paired samples and modality distributions, enabling classifiers trained on one modality to classify another. DeMIAN combines paired-sample matching with adversarial domain adaptation and reports improved classification, including state-of-the-art zero-shot results.
- High collection costs motivate using paired modal samples without labels and separately labeled single-modal data.Source: §1 · p. 1
- Modality-invariant representations place paired samples close together while making modality distributions similar.Source: §1 · p. 2
- Similar modality distributions can support decision boundaries that classify unseen-modality samples when a classifier is trained on another modality.Source: §1 · p. 2
- Prior multimodal methods and CCA exploit paired-sample relationships but do not address divergence between modality distributions.Source: §1 · p. 2
- DeMIAN jointly trains a generator to produce modality-invariant representations and a modality discriminator, while minimizing paired-sample distance and fooling the discriminator.Source: §1 · p. 2
- DeMIAN achieved better classification accuracy than state-of-the-art methods and state-of-the-art accuracy on benchmark zero-shot datasets.Source: §1 · p. 2
2. Related Works
Related work spans multimodal representation learning, domain adaptation, and zero-shot learning. DeMIAN combines paired-modality matching with distribution alignment to address modality differences while retaining the cross-modal relationships needed for shared representations.
2.1. Multimodal Learning
- 2.1. Multimodal Learning: Earlier models learn unified representations from paired modalities, whereas DeMIAN additionally aligns their distributions.Source: §2.1 · p. 2
2.2. Domain Adaptation
- 2.2. Domain Adaptation: Adversarial domain adaptation matches feature distributions, but its standard theory assumes the same modality and an ideal joint hypothesis.Source: §2.2 · p. 3
- 2.2. Domain Adaptation: DeMIAN addresses modality-specific properties and differing dimensions by finding a latent space where paired-sample distances are minimized.Source: §2.1 · p. 2, §2.2 · p. 3
2.3. Zero-Shot learning
- 2.3. Zero-Shot learning: The task predicts unseen target-domain classes from source-domain side information, and the paper integrates multimodal learning with domain adaptation for it.Source: §2.3 · p. 3
3. Proposed Method
DeMIAN learns common representations by combining paired-sample matching with modality-distribution alignment through adversarial training. Its formulation uses generators, a modality discriminator, and balanced optimization to support modality-invariant classification.
3. Proposed Method
- DeMIAN maps paired modalities into a common space while requiring similar distributions for their learned representations.The two requirements are paired-sample relationship preservation and distribution matching across modalities.Source: §3 · pp. 3–4
3.1. Generator to consider relationship between modalities
- The generators learn modality representations by minimizing a paired-sample distance such as Euclidean or cosine distance.This objective uses paired samples to preserve correspondence between modalities.Source: §3.1 · p. 4, §3 · p. 4
3.2. Modality-discriminator to make the distribution similar
- Adversarial training makes generators maximize the modality discriminator’s loss while the discriminator minimizes it to distinguish modalities.Gaussian samples are also supplied to the discriminator to discourage nondiscriminative representations that could trivially deceive it.Source: §3.2 · p. 4
3.3. Formulation of DeMIAN and its optimization
- The final objective balances multimodal matching and adversarial training with λ, selected using a dataset validation split.At the saddle point, generator parameters minimize modality-classification and paired-matching losses, while the implementation uses batch normalization and dataset-dependent optimization settings.Source: §3.3 · p. 5
4. Experiment
Across shared-representation and zero-shot experiments, DeMIAN generally outperformed comparison methods, including state-of-the-art results on SUN Attribute and CUB-200-2011.
4.1. MNIST
- On MNIST, DeMIAN matched modalities densely, produced more compact representations than MIAN, and retained comparable performance using only 1,000 labeled samples.Source: §4.1 · p. 6
4.2. Mir Flickr
- On Mir Flickr, DeMIAN outperformed existing methods for Image →Tag and Tag →Image while preserving source-modality performance and supporting joint-modality classification.Source: §4.2 · p. 6
4.3. SUN Attribute
- DeMIAN achieved state-of-the-art zero-shot accuracy on SUN Attribute and CUB-200-2011, with SUN’s score besting even an ensemble method.Source: Table 5 · p. 7, §4.3 · p. 7, §4.4 · p. 7
- On SUN shared-representation learning, DeMIAN exceeded 4% accuracy for Attribute →Image despite only 10 samples per class and random accuracy below 1%.Source: §4.3 · p. 7
4.4. CUB-200-2011
- On CUB-200-2011, DeMIAN improved state-of-the-art accuracy by about 3% and approached the current state of the art with only 0.5 of the labeled-sample ratio.Source: §4.4 · p. 7
4.5. CIFAR-10
- On CIFAR-10, DeMIAN performed better than Socher et al. [30] except for the cat-truck pair.Source: §4.5 · p. 8
5. Conclusion
The conclusion presents DeMIAN as an adversarial modality-invariant representation learner combining Domain Adaptation and multimodal learning. It reports better shared-representation and zero-shot performance, including state-of-the-art accuracy on CUB-200-2011 and SUN Attribute.
- DeMIAN combines Domain Adaptation and multimodal learning through adversarial training to learn modality-invariant representations.Source: §5 · p. 8
- DeMIAN achieved better performance in shared representation learning and zero-shot learning, with state-of-the-art accuracy on CUB-200-2011 and SUN Attribute.Source: §5 · p. 8