PaperlayerView exact revision on arXiv

Source-linked AI summary

CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

Yuxin Peng, Jinwei Qi, Yuxin Yuan

arXiv:1710.05106v2cs.MMcs.CVcs.LG

TL;DR

Different modalities have inconsistent distributions and representations, creating a heterogeneity gap that complicates correlation. CM-GANs models their joint distribution with cross-modal autoencoders and adversarial discrimination, and reports the best retrieval accuracy among compared methods across three datasets, including 0.559 versus 0.533 average MAP for bi-modal retrieval on XMediaNet.

  • Problem

    Inconsistent distributions and representations make heterogeneous modalities difficult to correlate, while prior GAN-based work mainly focuses on generating new data.

  • Method

    CM-GANs learns discriminative common representations by modeling cross-modal joint distributions with weight-sharing convolutional autoencoders and inter- and intra-modality adversarial discrimination.

  • Results

    CM-GANs achieves the best retrieval accuracy among compared methods across three datasets; on XMediaNet, average bi-modal MAP increases from 0.533 to 0.559.

  • Takeaways & Limitations

    The approach provides a GAN-based framework for learning more discriminative common representations across heterogeneous modalities for cross-modal retrieval.

Abstract

from arXiv · show

It is known that the inconsistent distribution and representation of different modalities, such as image and text, cause the heterogeneity gap that makes it challenging to correlate such heterogeneous data. Generative adversarial networks (GANs) have shown its strong ability of modeling data distribution and learning discriminative representation, existing GANs-based works mainly focus on generative problem to generate new data. We have different goal, aim to correlate heterogeneous data, by utilizing the power of GANs to model cross-modal joint distribution. Thus, we propose Cross-modal GANs to learn discriminative common representation for bridging heterogeneity gap. The main contributions are: (1) Cross-modal GANs architecture is proposed to model joint distribution over data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both of them beat each other to promote cross-modal correlation learning. (2) Cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form generative model. They can not only exploit cross-modal correlation for learning common representation, but also preserve reconstruction information for capturing semantic consistency within each modality. (3) Cross-modal adversarial mechanism is proposed, which utilizes two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make common representation more discriminative by adversarial training process. To the best of our knowledge, our proposed CM-GANs approach is the first to utilize GANs to perform cross-modal common representation learning. Experiments are conducted to verify the performance of our proposed approach on cross-modal retrieval paradigm, compared with 10 methods on 3 cross-modal datasets.

I. INTRODUCTION

Cross-modal data have heterogeneous distributions and representations, creating a gap that complicates correlation despite latent semantic relationships. CM-GANs address this by learning a discriminative common representation through joint-distribution modeling, weight-sharing autoencoders, and coupled intra- and inter-modality adversarial discrimination.

  • Heterogeneous multimedia modalities have inconsistent distributions and representations, creating a heterogeneity gap that complicates cross-modal correlation.
  • Latent cross-modal dependencies motivate mapping heterogeneous data into a shared space where semantic similarity can be measured directly.
  • Existing GAN-based work mainly generates new data in a single modality, whereas CM-GANs model multimodal joint distributions to correlate existing heterogeneous data.
  • CM-GANs use generative and discriminative models in a minimax game to learn cross-modal correlation through simultaneous joint-distribution modeling and relevance discrimination.
  • Cross-modal convolutional autoencoders share weights between modality pathways for common representations while decoders preserve within-modality reconstruction and semantic consistency.
  • Two discriminative models perform inter-modality and intra-modality discrimination, mutually boosting adversarial learning of more discriminative common representations.

II. RELATED WORKS

Related work covers traditional and deep-learning approaches for cross-modal correlation, alongside GANs primarily developed for generative applications. CM-GANs addresses the remaining need to model multimodal joint distributions rather than only one-way modality translation.

  • Cross-modal Correlation Learning Methods: Cross-modal correlation methods learn common representations that correlate heterogeneous modalities, using traditional or deep-learning-based models.Traditional approaches include projection and graph-regularization methods, while deep models use nonlinear correlation modeling.
  • Cross-modal Correlation Learning Methods: Traditional methods project different modalities into a common space by maximizing correlation or optimizing statistical relationships.Examples include CCA-like statistical correlation analysis and graph-regularized heterogeneous metric learning.
  • Cross-modal Correlation Learning Methods: Deep methods model cross-modal correlation with shared-layer autoencoders, joint belief networks, correspondence autoencoders, deep CCA, and hierarchical networks.These methods generally connect modality-specific subnetworks through shared or joint representations.
  • Generative Adversarial Networks: GANs combine a generative model that captures data distributions with a discriminative model trained adversarially to distinguish real from generated data.The minimax formulation reaches a global optimum when the generated distribution equals the data distribution.
  • Generative Adversarial Networks: Existing multimodal GAN applications mainly perform one-way generation, such as generating images from labels, sentences, or descriptions.These approaches use one-pathway networks and therefore do not model the joint distribution over multimodal inputs for correlation learning.

III. OUR CM-GANS APPROACH

CM-GANs uses paired image and text pathways to learn common representations for cross-modal similarity and retrieval. Its weight-sharing autoencoders preserve within-modality semantics, while two discriminator types jointly enforce intra- and inter-modality discrimination.

  • Formal Definition: CM-GANs learns common representations for image and text instances so cross-modal similarities and retrieval can be computed between modalities.The evaluation setting retrieves relevant text for an image query and images for a text query.
  • Generative Model: Cross-modal convolutional autoencoders form parallel image and text generative models with encoder and decoder layers.Encoders learn high-level semantic information and decoders generate reconstruction representations.
  • Generative Model: Decoder layers reconstruct high-level image and text representations to preserve semantic consistency within each modality.Image reconstruction targets the original image semantic representation, while text reconstruction targets its corresponding semantic representation.
  • Generative Model: Weight sharing in the final encoder layers correlates image and text common representations for corresponding pairs.The shared layers are designed so paired modalities produce similar common representations.
  • Discriminative Model: Two discriminator types perform intra-modality discrimination on reconstructions and inter-modality discrimination on common representations.The intra-modality models compare generated reconstructions with original representations, while inter-modality models distinguish matched and mismatched common representations.

C. Objective Functions

The objective functions jointly train two parallel GANs so generative models fit cross-modal relevance and reconstruction structure while discriminators distinguish both types of representations.

  • Objective Functions: CM-GANs generates original, common, and reconstruction representations for each modality to model inter-modality correlation and intra-modality reconstruction information.The generative models seek to fit the joint distribution while preserving modality-specific reconstruction information.
  • Objective Functions: Intra-modality discriminators distinguish original high-level representations from generated reconstruction representations for image and text.DI and DT provide discrimination within their respective modalities.
  • Objective Functions: Inter-modality discriminators distinguish common representations across modalities and reject mismatched same-modality representations.DCi treats image common representations as real and text or mismatched image representations as fake; DCt operates similarly for text.
  • Objective Functions: Generative and discriminative models are trained iteratively to learn more discriminative common representations for different modalities.The models compete in a minimax game during joint training of the two parallel GANs.
  • Objective Functions: The full objective minimizes over generative models and maximizes over intra- and inter-modality discriminators using two adversarial loss terms.LGAN1 represents intra-modality discrimination and LGAN2 represents inter-modality discrimination.

D. Cross-modal Adversarial Training Procedure

CM-GANs alternates discriminator and generator updates in a cross-modal adversarial procedure using matched and mismatched image-text pairs. Convolutional layers remain fixed, and the generator receives multiple updates to balance competing discrimination signals.

  • Training Procedure: The training procedure alternates generative and discriminative optimization while holding the opposite model fixed at each stage.The convolutional layers are also kept fixed because the focus is cross-modal correlation learning.
  • Discriminator Optimization: Intra-modality discriminator updates maximize correct separation between original high-level representations and generated reconstructions.The image pathway is described explicitly, with the text discriminator updated analogously.
  • Discriminator Optimization: Inter-modality discriminator updates distinguish common representations from the other modality and mismatched instances.The image pathway uses image common representations as real data and text or mismatched image representations as fake data.
  • Generator Optimization: The generative models are updated for K steps per iteration to balance multiple discriminator signals for intra- and inter-modality discrimination.The algorithm repeats these updates until CM-GANs converges and returns the optimized model.
  • Training Procedure: Each batch samples matching image-text pairs together with mismatching instances for cross-modal discrimination.The algorithm then generates representations through the image and text pathways before updating discriminators.

E. Implementation Details

The implementation uses Torch1, with the generative and discriminative model details introduced separately.

  • CM-GANs is implemented in Torch1, a scientific computing framework.
  • The following sections separately introduce the generative model and discriminative model implementation details.

1) Generative model:

The generative model uses two cross-modal convolutional autoencoder pathways to learn shared image-text representations while preserving modality-specific reconstruction information.

  • 1) Generative model:: Two parallel convolutional autoencoder pathways process image and text data to produce a common representation.The image pathway uses a 19-layer VGG-Net configuration, while the text pathway uses Word CNN.
  • 1) Generative model:: Weight sharing between the pathways’ second fully connected layers models correlation between modalities.Each pathway has two 1,024-unit fully connected layers, with batch normalization and ReLU activation.
  • 1) Generative model:: Decoder layers reconstruct each modality, preserving semantic consistency within image and text data.The decoders contain two fully connected layers on each pathway.

2) Intra-modality Discriminative model:

The intra-modality discriminator distinguishes original modality representations from their reconstructed counterparts using separate scoring networks.

  • 2) Intra-modality Discriminative model:: Each modality has a fully connected discriminator that maps an input feature vector to a sigmoid prediction score.The discriminator produces a single-value score for each modality.
  • 2) Intra-modality Discriminative model:: During discriminator training, original representations receive label 1 and reconstructed representations receive label 0.This labeling trains the model to distinguish original from reconstructed representations within each modality.

3) Inter-modality Discriminative model:

CM-GANs evaluates cross-modal common representations through intra- and inter-modality retrieval on three datasets, using MAP scores for comparison. The reported results show improved retrieval accuracy, while category difficulty and retrieval failures remain uneven.

  • 3) Inter-modality Discriminative model:: The inter-modality discriminator uses a two-pathway network whose inputs concatenate common and original representations.Each pathway contains two fully connected layers, with batch normalization, ReLU activation, and sigmoid scoring.
  • Experimental setup: Experiments compare CM-GANs with 10 state-of-the-art methods and five baselines across Wikipedia, Pascal Sentence, and XMediaNet datasets.The evaluation uses image-text data, including 40,000 image and text instances from XMediaNet.
  • Retrieval evaluation: Cross-modal retrieval includes image-to-text, text-to-image, image-to-all, and text-to-all tasks evaluated with mean average precision.MAP combines ranking information and precision over returned results.

C. Compared Methods

The study compares CM-GANs with traditional and deep learning methods for cross-modal common-space and correlation learning. The baselines use projection, correlation, reconstruction, semantic matching, or multi-level deep representations.

  • Traditional methods: CCA, CFA, KCCA, JRL, and LGCFL represent traditional approaches for learning a shared cross-modal space.CCA maximizes correlation, CFA minimizes projection error, KCCA extends CCA with kernels, JRL uses semi-supervised and sparse regularization, and LGCFL uses local-group priors.
  • Deep learning based methods: Corr-AE jointly models cross-modal correlation and reconstruction errors through subnetworks linked at the code layer.
  • Deep learning based methods: DCCA maximizes correlation between two separate subnetworks using an objective similar to CCA.
  • Deep learning based methods: CMDN jointly models intra-modality and inter-modality correlation in separate and common representation-learning stages.
  • Deep learning based methods: Deep-SM uses convolutional networks for image representation and semantic matching, while CCL combines intra-modality and inter-modality correlation with multi-grained multi-task learning.

D. Comparisons with 10 State-of-the-art Methods

CM-GANs is evaluated against 10 state-of-the-art methods on three cross-modal retrieval datasets. It achieves the strongest reported retrieval performance, while category-level failures remain associated with visual and semantic ambiguity.

  • Comparisons with 10 State-of-the-art Methods: CM-GANs achieves the best retrieval accuracy among all compared methods on bi-modal and all-modal retrieval across the evaluated datasets.On XMediaNet, average bi-modal MAP rises from 0.533 to 0.559; similar trends are reported for Pascal Sentence and Wikipedia.
  • Comparisons with 10 State-of-the-art Methods: 0.559 average MAP is reported for CM-GANs on bi-modal retrieval on XMediaNet, improving from 0.533.
  • Comparisons with 10 State-of-the-art Methods: Most deep learning methods outperform traditional methods, with CCL strongest among compared methods and LGCFL and JRL strongest among traditional methods.
  • Experimental Analysis: CM-GANs combines joint-distribution modeling, weight-sharing and semantic constraints, and inter-modality and intra-modality discrimination.
  • Experimental Analysis: Failure cases mainly arise from small visual variance or confusing text, while high-level semantics and small objects produce category-specific confusion.CM-GANs reduces failure cases relative to CCL and LGCFL and achieves the best accuracy on most reported categories.

1) Performance of generative model:

Ablation experiments examine the contributions of the generative constraints, discriminative components, and adversarial training in CM-GANs. The reported results support complementary benefits from these components for cross-modal retrieval.

  • Performance of discriminative model:: Intra-modality discrimination complements inter-modality discrimination and preserves semantic consistency within each modality.The comparison uses the “CM-GANs only inter” baseline, which omits intra-modality discrimination.
  • Performance of generative model:: Weight-sharing and semantic constraints make similar contributions to final retrieval accuracy in the generative model.Weight-sharing handles cross-modal correlation, while semantic constraints preserve semantic consistency between modalities.
  • Performance of adversarial training:: CM-GANs obtains higher average bi-modal MAP than CM-GANs-CAE on three datasets.CM-GANs-CAE trains the generative model without adversarial training.
  • Performance of generative model:: The baseline results attribute gains to cross-modal and semantic constraints, complementary intra-modality discrimination, and adversarial training.
Loading 1710.05106v2…