PaperlayerView exact revision on arXiv

Source-linked AI summary

CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network

Yuxin Peng, Jinwei Qi, Xin Huang, Yuxin Yuan

arXiv:1704.02116v4cs.MM

TL;DR

Existing DNN-based cross-modal retrieval methods overlook inter-modality context, balance correlation constraints inadequately, and ignore fine-grained patch clues. CCL addresses these gaps with joint correlation optimization, multi-task learning, and multi-grained fusion, and reports the best performance against 13 state-of-the-art methods on 6 datasets.

  • Problem

    Existing DNN methods model limited correlation, use single-loss regularization, and omit complementary fine-grained patch clues in two-stage cross-modal retrieval.

  • Method

    CCL jointly optimizes intra-modality and inter-modality correlation, adaptively balances semantic and pairwise constraints, and fuses coarse-grained instances with fine-grained patches.

  • Results

    CCL achieves the best performance when compared with 13 state-of-the-art methods on 6 widely-used cross-modal datasets.

  • Takeaways & Limitations

    CCL makes cross-modal correlation more precise by integrating complementary context from modalities, semantic constraints, pairwise similarities, instances, and patches.

Abstract

from arXiv · show

Cross-modal retrieval has become a highlighted research topic for retrieval across multimedia data such as image and text. A two-stage learning framework is widely adopted by most existing methods based on Deep Neural Network (DNN): The first learning stage is to generate separate representation for each modality, and the second learning stage is to get the cross-modal common representation. However, the existing methods have three limitations: (1) In the first learning stage, they only model intra-modality correlation, but ignore inter-modality correlation with rich complementary context. (2) In the second learning stage, they only adopt shallow networks with single-loss regularization, but ignore the intrinsic relevance of intra-modality and inter-modality correlation. (3) Only original instances are considered while the complementary fine-grained clues provided by their patches are ignored. For addressing the above problems, this paper proposes a cross-modal correlation learning (CCL) approach with multi-grained fusion by hierarchical network, and the contributions are as follows: (1) In the first learning stage, CCL exploits multi-level association with joint optimization to preserve the complementary context from intra-modality and inter-modality correlation simultaneously. (2) In the second learning stage, a multi-task learning strategy is designed to adaptively balance the intra-modality semantic category constraints and inter-modality pairwise similarity constraints. (3) CCL adopts multi-grained modeling, which fuses the coarse-grained instances and fine-grained patches to make cross-modal correlation more precise. Comparing with 13 state-of-the-art methods on 6 widely-used cross-modal datasets, the experimental results show our CCL approach achieves the best performance.

I. INTRODUCTION

Cross-modal retrieval addresses heterogeneous multimedia data by learning representations that support similarity measurement across modalities. CCL targets limitations in existing two-stage DNN methods through joint correlation learning, multi-task constraints, and fusion of instances with patches.

  • Motivation: Cross-modal retrieval enables queries in one modality to retrieve relevant multimedia results in other modalities, but heterogeneous distributions make cross-modal similarity difficult to measure.The paper contrasts this with single-modal retrieval, which returns results in the same modality as the query.
  • Existing framework: Existing DNN-based methods use two stages: separate representation learning followed by common representation learning through cross-modal correlation.Their first-stage representations mainly model intra-modality correlation, while second-stage learning commonly uses single-loss regularization.
  • CCL contributions: CCL jointly optimizes intra-modality and inter-modality correlation in the first stage to preserve complementary context for separate representation learning.The approach uses multi-level association so cross-modal hints contribute during representation learning.
  • CCL contributions: CCL uses multi-task learning to adaptively balance intra-modality semantic category constraints and inter-modality pairwise similarity constraints.The strategy is designed to exploit their intrinsic relevance rather than relying on a single loss.
  • CCL contributions: CCL fuses coarse-grained instances with fine-grained patches through a multi-pathway network to make cross-modal correlation more precise.This addresses the omission of complementary patch clues in methods that use only original instances.

II. RELATED WORKS

DNN-based cross-modal retrieval commonly uses two stages: separate modality representations followed by common-representation learning. Existing approaches differ in architecture and correlation modeling, but often omit inter-modality information, multi-task constraints, or fine-grained patches.

  • Traditional and DNN-based methods: Traditional methods include CCA, semi-supervised learning, and graph regularization, while DNN methods model complex nonlinear cross-modal correlation.CCA learns a common space maximizing correlation between paired modalities; graph-based methods learn projection matrices with regularization.
  • DNN-based methods: DNN-based methods generally generate separate representations first and then learn a common representation by exploiting cross-modal correlation.Representative models include Multimodal DBN, Bimodal AE, Corr-AE, and CMDN.
  • Limitations: Existing methods often model only intra-modality correlation during separate representation learning, ignoring complementary inter-modality context.This limitation is identified for methods such as Multimodal DBN and Corr-AE.
  • Limitations: Single-loss regularization can overlook the intrinsic relevance between intra-modality and inter-modality correlation in common-representation learning.The paper motivates multi-task learning as a way to address this limitation.
  • Limitations: DNN-based methods generally consider original instances, whereas CCL additionally models complementary fine-grained patches through multi-grained fusion.CCL uses a multi-pathway network to fuse coarse-grained instances and fine-grained patches.

III. OUR CCL APPROACH

CCL is introduced as a hierarchical approach that models cross-modal correlation across original instances and patches. Its formal setup uses paired image and text modalities with associated semantic labels and correspondence.

  • Approach overview: CCL simultaneously models intra-modality and inter-modality correlation while using coarse-grained instances and fine-grained patches.The approach is designed to learn more precise cross-modal correlation.
  • Data formulation: The multimodal dataset contains m image instances and n text instances, with each instance represented by features and an associated label.The provided formal definition introduces image and text variables and their labels.
  • Data formulation: Pairwise correspondence denotes image-text instances that co-exist to describe relevant semantics.This correspondence provides the paired cross-modal structure used by CCL.

A. First Learning Stage: Multi-grained Fusion with Joint Optimization

The first learning stage obtains separate representations by processing original instances and patches through modality-specific pathways. Joint optimization combines reconstruction and cross-modal correlation learning.

  • First learning stage: A multi-pathway network obtains separate representations from original instances and patches while jointly capturing intra-modality and inter-modality correlation.The two types of information are modeled in parallel during the first stage.
  • Coarse-grained learning: Two modality-specific DBNs model image and text feature distributions, using Gaussian RBMs for images and Replicated Softmax models for text.The DBN outputs preserve modality-specific characteristics with high-level semantic information.
  • Joint optimization: The two pathways are linked at the top code layer to jointly optimize reconstruction learning error and cross-modal correlation learning error.This differs from CMDN’s use of independent networks for intra-modality and inter-modality modeling.
  • Joint optimization: The reconstruction loss minimizes L2 distance between each modality’s input and reconstruction, while the correlation loss minimizes L2 distance between different modalities.Together they produce coarse-grained representations containing both correlation types.
  • Fine-grained learning: Fine-grained processing divides images and text into patches, then applies analogous two-pathway DBN modeling and joint optimization.Selective search extracts image regions, while text is segmented into paragraphs, sentences, or words.

2) Fine-grained learning with patches:

CCL learns fine-grained representations from image and text patches, then fuses them with original-instance representations. This multi-grained fusion preserves complementary intra-modality and inter-modality information.

  • Fine-grained learning with patches: Patch representations are obtained by linking image and text pathways at the code layer and jointly minimizing reconstruction and correlation losses.The losses have definitions analogous to those used for original instances.
  • Fine-grained learning with patches: Fine-grained patch representations preserve intra-modality and inter-modality correlation across image and text patches.These representations provide complementary information beyond original instances.
  • Multi-grained fusion: A joint RBM fuses coarse-grained original-instance representations with fine-grained patch representations for each modality.The two intermediate representation types are treated as inputs to the joint distribution.
  • Multi-grained fusion: The resulting separate image and text representations capture intrinsic correlation and complementary information from original instances and patches.The fusion process is applied to both modalities.

B. Second Learning Stage: Multi-task Cross-modal Correlation Learning

The second learning stage uses multi-task learning to jointly model intra-modality semantic constraints and inter-modality pairwise similarity, producing common representations for retrieval.

  • Multi-task learning: Two loss branches model intra-modality semantic category constraints and inter-modality pairwise similarity constraints.The former enhances semantic discrimination, while the latter captures cross-modal correlation for more accurate common representations.
  • Inter-modality constraints: Contrastive loss models both similar and dissimilar image-text pairs, minimizing distances for same-label pairs and separating dissimilar pairs.Existing methods are described as focusing mainly on similar pairwise constraints.
  • Common representation: The image and text pathways use nonlinear mappings to convert separate representations into final common representations.Each pathway consists of three fully connected layers, with margin parameter α used in the pairwise-learning formulation.
  • Intra-modality constraints: An n-way softmax classification branch assigns each modality's data to one of n categories to exploit intra-modality semantic information.Minimizing cross-entropy enhances the semantic discrimination ability of the common representation.
  • Retrieval: The resulting representations are obtained from the last fully connected layer and support cross-modal similarity measurement.Image and text representations are extracted from original instances and patches before applying a distance metric such as cosine distance.

IV. EXPERIMENTS

The experiments evaluate CCL against 13 state-of-the-art methods on 6 widely used cross-modal datasets, including analyses of components, parameters, networks, and execution time.

  • Experimental design: CCL is tested on 6 widely used cross-modal datasets against 13 state-of-the-art methods.The study also includes network and parameter analyses, execution-time evaluation, and baseline experiments for individual components.

A. Datasets

The experiments use six cross-modal datasets spanning image-text pairs, labels, tags, articles, sentences, and large-scale web imagery, with original instances and patches represented using matched features.

  • Dataset composition: Wikipedia contains 2,866 image-text pairs from 10 categories, split into 2,173 training, 231 validation, and 462 testing pairs.The dataset is randomly split following prior work.
  • Dataset composition: NUS-WIDE provides about 70,000 image-text pairs selected from images belonging exclusively to one of 10 largest categories.The source dataset contains about 270,000 images with tags across 81 classes.
  • Dataset composition: NUS-WIDE-10K contains 10,000 image-text pairs evenly selected from 10 categories, with 8,000 training, 1,000 testing, and 1,000 validation pairs.Pascal Sentence is also introduced as a 1,000-image dataset evenly categorized into 20 categories.
  • Dataset composition: Flickr-30K contains 31,784 images annotated with five sentences each, while MS-COCO contains 123,287 images with annotated sentences.Both datasets use crowdsourced sentence annotations and predefined training, validation, and testing splits.
  • Patch construction: Image patches are generated with selective search, filtered by an IoU threshold of 0.7, and the 10 largest non-overlapping patches are selected.Text instances are segmented according to dataset format, with tags generally divided into four patches.
  • Feature representation: The same feature type and dimensionality are extracted separately for each original instance and its patches.Features include hand-crafted representations and 4,096-dimensional VGGNet fc7 CNN features used across the six datasets.

D. Details of the Network

CCL uses hierarchical networks for two learning stages, combines original instances with patches, and evaluates bi-modal and all-modal retrieval using dataset-appropriate metrics.

  • First learning stage: The first learning stage uses separate two-layer DBNs for original images, text, and their patches, followed by joint optimization of intra- and inter-modality correlations.Representations of original instances and patches are fused with joint RBMs and further optimized using a feed-forward network with softmax loss.
  • Second learning stage: The second learning stage is a three-layer fully connected multi-task network with separate loss branches for intra-modality and inter-modality learning.The fully connected layers use 1,024 dimensions for the Wikipedia configuration.
  • Retrieval tasks: CCL performs bi-modal retrieval between image and text and all-modal retrieval that returns both modalities for either query type.Flickr-30K and MS-COCO additionally support image annotation and image retrieval tasks.
  • Compared methods: The experiments compare CCL with 13 state-of-the-art methods, supplementing unavailable published results with authors' source-code implementations.The study reports retrieval results across the selected datasets and tasks.
  • Evaluation: MAP evaluates retrieval on four labeled datasets, while Recall@1, Recall@5, and Recall@10 are reported for Flickr-30K and MS-COCO.Precision-recall and precision-scope curves are also used for comprehensive evaluation on NUS-WIDE.

F. Comparison with State-of-the-art Methods

CCL outperforms the compared methods across six cross-modal datasets and both bi-modal and all-modal retrieval settings. Its advantage is attributed to joint correlation modeling, multi-task learning, and fusion of coarse- and fine-grained information.

  • Wikipedia results: CCL achieves the best results in bi-modal and all-modal retrieval on Wikipedia with both hand-crafted and CNN features.The paper also reports fewer retrieval failure cases than CMDN, while noting confusion in difficult high-level semantic categories.
  • Cross-dataset results: CCL maintains the best performance across the other five datasets with both hand-crafted and CNN features.Precision-recall and precision-scope curves on NUS-WIDE further verify the approach’s effectiveness.
  • Overall comparison: CCL keeps an advantage over 13 compared methods on six datasets, with similar result trends across experiments.The compared methods include traditional correlation-based approaches and DNN-based methods such as DCCA and CMDN.
  • Sources of advantage: CCL improves over competing methods by jointly optimizing intra- and inter-modality correlations, balancing semantic and pairwise constraints, and fusing patches with original instances.Compared methods generally use only original modality instances, single-loss regularization, or omit inter-modality correlation during separate representation learning.

G. Experimental Analysis

Additional experiments examine CCL’s convergence, training settings, parameter size, execution time, and component behavior. These analyses indicate rapid convergence, comparable testing time in one setting, and benefits from multi-grained inputs.

  • Convergence: CCL converges within 2K iterations on the Wikipedia and NUS-WIDE datasets.The convergence plots show a downtrend in loss value and are used to verify convergence and efficiency.
  • Parameter size: CCL originally has about 21K parameters, compared with about 17K for CMDN and 10K for Corr-AE.The paper performs comparable parameter-space experiments by reducing layers and hidden units in CCL and CMDN.
  • Execution time: Testing time is almost the same for all methods on the small-scale Wikipedia dataset.On large-scale NUS-WIDE, execution-time differences mainly reflect differing common-representation dimensions and distance-measurement costs.
  • Multi-grained analysis: CCL integrates coarse-grained instances and fine-grained patches more accurately than using either information type alone.The ablations compare CCL using only coarse-grained instances or only fine-grained patches against the full multi-grained model.

3) Baseline Experiments:

Baseline experiments show that combining intra- and inter-modality correlation and fusing coarse- and fine-grained information improves CCL’s retrieval accuracy. The conclusion identifies more precise fine-grained representation learning as future work.

  • Multi-grained fusion: Fusing coarse-grained instances with fine-grained patches produces better accuracy than either single-grained input.CCL removes the fusion component in the single-grained baselines while keeping the remaining network components unchanged.
  • Correlation learning: Jointly learning intra- and inter-modality correlations achieves higher MAP scores than learning either correlation alone.The result indicates that the two information types provide complementary context for separate representation learning.
  • Interpretation: Inter-modality correlation provides complementary context to intra-modality correlation for learning separate representations.The learned representations can capture additional hints for common representation learning.
  • Conclusion: CCL combines joint correlation optimization, multi-task constraints, and multi-grained fusion in a hierarchical cross-modal retrieval approach.The paper reports effectiveness against 13 state-of-the-art methods on six widely used cross-modal datasets.
  • Future work: Future work will target better fine-grained representations through more effective and precise segmentation methods.The authors also plan to incorporate semi-supervised regularization to use unlabeled data.
Loading 1704.02116v4…