PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Robust Visual-Semantic Embeddings

Yao-Hung Hubert Tsai, Liang-Kang Huang, Ruslan Salakhutdinov

arXiv:1703.05908v2cs.CVcs.CLcs.LG

TL;DR

Existing joint image-text embedding methods largely rely on supervised paired data, leaving robust cross-domain representation learning with unlabeled data as an open challenge. The paper introduces an end-to-end framework combining auto-encoders, Maximum Mean Discrepancy, and unsupervised-data adaptation inference. It reports consistent improvements over other methods across zero- and few-shot recognition and retrieval experiments.

  • Problem

    Existing joint image-text embedding methods use mainly supervised information from paired images and textual attributes, while robust cross-domain representations remain challenging.

  • Method

    ReViSE combines auto-encoders for visual and textual features with cross-modality distribution matching and unsupervised-data adaptation inference.

  • Results

    ReViSE consistently outperforms other methods across zero- and few-shot recognition and retrieval experiments from inductive to transductive settings.

  • Takeaways & Limitations

    Combining supervised and unsupervised objectives can produce more useful joint embeddings from labeled and unlabeled visual and textual data.

  • Takeaways & Limitations

    Adding contractive loss to the textual auto-encoder provides little benefit, possibly because CUB has only 200 textual features compared with 11,786 visual features.

Abstract

from arXiv · show

Many of the existing methods for learning joint embedding of images and text use only supervised information from paired images and its textual attributes. Taking advantage of the recent success of unsupervised learning in deep neural networks, we propose an end-to-end learning framework that is able to extract more robust multi-modal representations across domains. The proposed method combines representation learning models (i.e., auto-encoders) together with cross-domain learning criteria (i.e., Maximum Mean Discrepancy loss) to learn joint embeddings for semantic and visual features. A novel technique of unsupervised-data adaptation inference is introduced to construct more comprehensive embeddings for both labeled and unlabeled data. We evaluate our method on Animals with Attributes and Caltech-UCSD Birds 200-2011 dataset with a wide range of applications, including zero and few-shot image recognition and retrieval, from inductive to transductive settings. Empirically, we show that our framework improves over the current state of the art on many of the considered tasks.

1. Introduction

Learning comprehensive representations across visual and textual modalities remains challenging, especially when zero- and few-shot tasks require transferring knowledge through textual attributes. ReViSE addresses this by combining supervised and unsupervised learning to construct joint visual-semantic embeddings from labeled and unlabeled data.

  • Approach: ReViSE learns robust joint embeddings across visual and textual modalities in an end-to-end framework for zero- and few-shot settings.
  • Motivation: Zero-shot recognition and retrieval use textual attributes to transfer knowledge to novel classes without training labels.Few-shot learning similarly operates with only a few labeled examples in target categories.
  • Approach: Auto-encoders learn representations from labeled and unlabeled data using reconstruction objectives rather than directly aligning visual and textual inputs.
  • Approach: Cross-modality distribution matching aligns the distributions of visual and textual codes, improving recognition and retrieval performance.
  • Results: Unsupervised-data adaptation inference improves accuracy for ReViSE and many existing models on unlabeled data.
  • Results: ReViSE improves recognition and retrieval performance on AwA and CUB under zero- and few-shot settings.

2. Related Work

Prior work develops multimodal embeddings and zero- or few-shot methods by combining visual representations with textual or other side information. ReViSE is motivated by evidence that unsupervised learning can benefit cross-modal feature learning.

  • Multimodal Embeddings: Multimodal embedding research targets representations spanning visual and textual domains for tasks such as classification and retrieval.
  • Zero and Few-Shot Learning: Zero-shot learning recognizes novel classes without examples by relying on side information such as textual descriptions, whereas few-shot learning uses one or a few examples.
  • Visual-Semantic Embeddings: Existing visual-semantic methods commonly transform image and text representations and learn relations or alignments for recognition and retrieval.
  • Unsupervised Multimodal Learning: Previous multimodal studies provide evidence that unsupervised learning tasks can improve cross-modal feature learning and motivate combining supervised and unsupervised objectives.

3. Proposed Method

The proposed method learns joint visual-semantic embeddings by combining supervised alignment with auto-encoder reconstruction, cross-modality distribution matching, and unsupervised-data adaptation. It incorporates labeled and unlabeled data for zero- and few-shot recognition and retrieval.

  • 3.2. Reconstructing Features from Auto-Encoder: Auto-encoders reconstruct visual and textual features, with a contractive auto-encoder used to learn more robust visual codes.The visual and textual auto-encoders are added after pretrained networks process the inputs.
  • 3.3. Cross-Modality Distributions Matching: Maximum Mean Discrepancy matches the distributions of visual and textual hidden representations to narrow the information gap across modalities.The method uses a Gaussian kernel for the MMD criterion.
  • 3.4. Learning: Supervised binary prediction loss aligns labeled training images with their corresponding textual attributes.The authors report that this simple binary prediction loss performs best empirically among the considered alternatives.
  • 3.4. Learning: Unsupervised-data adaptation inference uses empirical predictions to incorporate unlabeled images and attributes into learning.The authors describe it as a self-reinforcing strategy, with λ = 1.0 working well for many considered methods.
  • 3. Proposed Method: The model combines supervised and unsupervised objectives to learn joint embeddings from labeled and unlabeled visual and textual data.The unsupervised objective can also act as a regularizer for more robust representations.

4. Experiments

Experiments evaluate ReViSE on AwA and CUB across zero- and few-shot recognition and retrieval, using visual features with multiple textual-attribute types. Results show benefits from unsupervised objectives, cross-modal distribution matching, and unsupervised-data adaptation, although gains vary by setting.

  • Experimental Setup: ReViSE is evaluated on AwA and CUB using zero- and few-shot recognition and retrieval tasks with att, w2v, and glo textual attributes.CUB is fine-grained, whereas AwA represents more general concepts.
  • Experimental Setup: The ablation variants progressively add unsupervised objectives and unlabeled training data to the supervised ReViSEa baseline.ReViSEb adds reconstruction and MMD objectives for labeled data; ReViSEc additionally incorporates unlabeled training data.
  • Ablation Results: 2.6% recognition and 0.4% retrieval improvements result from adding unlabeled training data, comparing ReViSEb with ReViSEc.The comparison indicates that the architecture benefits from unlabeled visual and textual training data.
  • Ablation Results: 5.0% average recognition and 5.8% average retrieval improvements result from unsupervised-data adaptation, comparing ReViSEc with ReViSE.The authors additionally report significant improvement for unlabeled training images Vut.
  • Zero-Shot Results: ReViSE improves zero-shot recognition by at least 17% over methods without unsupervised objectives and 7.5% over DeViSE* and CMT*, while achieving 14.1% and 21.5% retrieval gains over the corresponding baselines.On AwA, ReViSE attains comparable top-1 accuracy with att, w2v, and glo attributes: 93.4%, 93.5%, and 92.2%.
  • Component Analysis: 68.1% versus 66.8% average performance shows that minimizing MMD between visual and textual codes improves the learned embeddings.Removing the contractive loss instead reduces performance from 66.8% to 65.8%.
  • Few-Shot Results: Transductive few-shot results generally favor ReViSE, but its average retrieval advantage over DeViSE* narrows from 15.5% in zero-shot experiments to 9.3% in few-shot experiments.Adding three labeled test-class images improves performance for every method, including a reported 32.3% retrieval improvement for ReViSE on CUB with w2v attributes.

5. Conclusion

The paper augments supervised learning with unsupervised techniques for visual-textual joint embeddings and evaluates the method across datasets, tasks, and learning settings. It consistently outperforms other methods, with substantial improvements in some cases.

  • ReViSE augments a typical supervised formulation with unsupervised techniques to learn joint embeddings of visual and textual data.
  • The method is evaluated on general and fine-grained image classification datasets across zero- and few-shot recognition and retrieval tasks.
  • ReViSE consistently outperforms other methods across inductive and transductive settings, substantially improving performance in some cases.

6. Network Design

The network design uses pretrained visual features and normalized textual attributes, then applies fully connected visual and textual encoders with training procedures for unsupervised-data adaptation.

  • The architecture uses 1024-dimensional top-layer GoogLeNet activations from ImageNet without fine-tuning as input image features.
  • Textual attributes are preprocessed with standard l2 normalization before entering the network.
  • The visual encoder is a two-hidden-layer fully connected neural network with architecture dv1−dv2−dc, where dv1 = 1024 and dv2 = 500.
  • During the first 100 training iterations, λ = 0 disables unsupervised-data adaptation while the inferred labels are still updated.
  • From iteration 101, λ is set to 0.1 or 1.0 by cross-validation, and training typically converges within 2000 to 5000 iterations.
  • ReViSE is implemented in TensorFlow using Adam, minibatches of 1024, and tanh activation functions.

7. Parameters Choice

The experiments fix several architecture parameters, select β and λ by cross-validation, and study how the unsupervised-objective weight α affects performance on CUB.

  • The experiments fix α = 1.0, γ = 0.1, and κ = 32.0, then use cross-validation to choose β and λ from {0.1, 1.0}.
  • Figure 7 presents the designed ReViSE architecture, and Table 8 reports the selected β and λ values.
  • The parameter study uses CUB with att attributes to examine the contribution of unsupervised information through α.
  • Performance increases as α rises from 0.1 to 1.0, while the study also evaluates larger values from 1.0 to 5.0.

8. Precision-Recall Curve

The section presents the precision-recall curve for zero-shot retrieval on CUB using att attributes.

  • Figure 10 shows the precision-recall curve for zero-shot retrieval results on CUB with att attributes.

9. MMD Distance

The method aligns visual and textual codes using MMD, and the transductive CUB experiments compare this alignment across iterations, LMMD settings, and retrieval precision-recall.

  • MMD distance measures the distribution mismatch between visual and textual codes in the transductive zero-shot CUB experiment.The experiment evaluates ReViSE with LMMD against ReViSE† without LMMD on test codes.
  • ReViSE maintains a lower MMD distance than ReViSE† across the reported iterations.The figure compares the method with and without LMMD using red and blue curves, respectively.
  • The α sensitivity experiments cover transductive zero-shot recognition and retrieval on CUB using att attributes.α is varied over two scales: 0.1 to 1.0 and 0.5 to 5.0.
  • Figure 10 compares precision-recall curves for zero-shot retrieval on CUB using human-annotated attributes as class textual attributes.

10. Remarks on Contractive Loss

Adding contractive loss to the textual auto-encoder provides little benefit, possibly because the textual feature dimensionality is limited relative to the visual feature dimensionality.

  • Adding contractive loss to the textual auto-encoder does not provide much benefit.
  • The proposed explanation is that CUB has 200 textual features but 11,786 visual features.

11. Comparing with recent state-of-the-art methods

The paper compares its method with other approaches for inductive and transductive zero-shot learning, focusing in the main paper on deep-embedding methods.

  • The main paper focuses on comparisons with deep-embedding methods.
  • Table 9 compares other methods under both inductive and transductive zero-shot learning settings.
  • SMSESZSL uses ESZSL for initialization.
Loading 1703.05908v2…