Source-linked AI summary
Linking Image and Text with 2-Way Nets
Aviv Eisenschtat, Lior Wolf
TL;DR
The paper addresses matching paired vectors from two data sources beyond linear or directly CCA-optimized approaches. It introduces a tied bidirectional neural network using Euclidean loss, with training modifications for regression challenges, and reports significant advantages over literature methods across correlation-analysis benchmarks.
Problem
Matching vectors from two sources requires methods that extend beyond linear CCA and address the limitations of directly optimizing correlation-based objectives in deep models.
Method
The method uses a tied two-way neural architecture with Euclidean loss, decorrelation and variance-preserving regularization, modified batch normalization, and tied dropout.
Results
The method significantly outperforms classical, modern variant, and recent deep correlation methods across the reported matching experiments.
Takeaways & Limitations
Euclidean loss can support correlation maximization in a bidirectional matching network when combined with the paper's training modifications.
Takeaways & Limitations
The paper identifies low batch-normalization biases in most trained networks and suggests they may be removable; it also leaves tied two-way matching across domains as future work.
Abstract
from arXiv · showhide
Linking two data sources is a basic building block in numerous computer vision problems. Canonical Correlation Analysis (CCA) achieves this by utilizing a linear optimizer in order to maximize the correlation between the two views. Recent work makes use of non-linear models, including deep learning techniques, that optimize the CCA loss in some feature space. In this paper, we introduce a novel, bi-directional neural network architecture for the task of matching vectors from two data sources. Our approach employs two tied neural network channels that project the two views into a common, maximally correlated space using the Euclidean loss. We show a direct link between the correlation-based loss and Euclidean loss, enabling the use of Euclidean loss for correlation maximization. To overcome common Euclidean regression optimization problems, we modify well-known techniques to our problem, including batch normalization and dropout. We show state of the art results on a number of computer vision matching tasks including MNIST image matching and sentence-image matching on the Flickr8k, Flickr30k and COCO datasets.
1. Introduction
The paper frames computer vision as linking visual data with multidimensional external sources and introduces a tied, bidirectional network that uses Euclidean loss instead of directly optimizing correlation loss. Training modifications address Euclidean regression challenges, and experiments report advantages over classical and recent methods.
- Computer vision includes matching visual data with multidimensional sources such as audio, text, or images from different views.
- The proposed architecture maps one source to another and back through tied neural-network channels using Euclidean loss.
- The method adds mid-way loss, decorrelation regularization, modified batch normalization, scale regularization, tied dropout, and high-dimensional-data handling.
- The resulting method is described as general and robust, with clear advantages over classical and recent methods in extensive experiments.
2. Previous work
Prior work develops CCA through linear, kernel, and deep variants, while autoencoder-based approaches connect multiple views through shared or aligned representations. This paper instead uses a bidirectional cross-view architecture and adapts deep-learning techniques to make Euclidean-loss training effective.
- CCA projects two views into a common space to maximize correlation, while its regularized, kernel, and randomized variants extend the basic method.
- Deep CCA processes paired inputs through two neural pipelines and compares their outputs with the CCA loss to address limitations of classical methods.
- Unlike earlier image-text extensions, the paper presents a novel deep model for matching images and text.
- Autoencoder-based methods encode and decode inputs, whereas this model maps one view to the other in both directions rather than reconstructing each view.
- The model successfully uses L2 loss by adapting dropout, batch normalization, and leaky ReLUs to the matching problem.
- Dropout reduces neuron co-adaptation by stochastically eliminating groups of neurons during training.
- Batch normalization stabilizes training by scaling hidden-layer outputs toward zero norm and unit variance.
- The paper uses a fixed-leakiness ReLU variant that reduces the effect of zeroing negative activations without learning the leakiness parameter.
3. The Network Model
The 2-way net uses tied reconstruction channels to map two views into one another and align their middle representations with Euclidean losses. The paper links this objective to correlation maximization and adds normalization, regularization, activation, and dropout mechanisms to preserve informative variance and stable matching.
- Basic Architecture: The 2-way net uses two reconstruction channels with shared weights to transform each view into the other.The channels define reconstructions x̃ = Ĥ(y) and ỹ = H(x), using transposed tied layers with separate biases.
- Basic Architecture: A mid-network Euclidean loss aligns corresponding hidden representations, while reconstruction losses align each output with its original view.The model selects the middle position j = ⌈k/2⌉ and compares Hj(x) with Ĥj(y).
- Correlation Objective: Minimizing L2 loss while increasing each neuron’s variance maximizes the sum of correlations between matching hidden representations.The paper establishes this connection through lemmas relating correlation to Euclidean distance and activation variance.
- Activation and Normalization: For correlation 0.6, the matching-neuron intersection contains about 35% of neurons, motivating leaky ReLU and batch normalization after nonlinearities.The paper reports that asymmetric activation scaling can otherwise increase L2 distance and reduce training efficiency.
- Variance Control: Batch normalization and gamma regularization prevent hidden-output variance from collapsing during regression training.The gamma penalty encourages high variance and is reported to force the network away from low-variance solutions.
- Tied Dropout: Conventional dropout creates artificial mismatch and shrinking effects, so tied dropout applies the same random mask to matching hidden layers.At dropout probability p = 0.5, half of gradients arise from matches silent on exactly one side.
4. Experiments
Experiments evaluate 2WayNet on CCA benchmarks, image-sentence retrieval, and architectural ablations. The method outperforms prior methods broadly, while batch normalization, variance regularization, tied dropout, and leaky ReLU materially affect optimization and correlation.
- Comparison with published results: 2WayNet outperforms six CCA variants by a large margin on XRMB and regains half the remaining correlation on MNIST.MNIST’s literature results are already close to the maximal value of 50.
- Image annotation and search: 2WayNet achieves best performance across almost all image-sentence matching scores on Flickr8k, Flickr30k, and COCO, especially for image annotation r@1.The authors report large improvements for image annotation across all three datasets.
- Ablation analysis: Batch normalization has a profound effect: post-ReLU placement and 1/γ regularization increase hidden variance, stabilize training, and improve correlation.Without batch normalization, higher learning rates prevented convergence; the ablation covers Flickr8k, Flickr30k, MNIST, and XRMB.
- Ablation analysis: Tied dropout outperforms conventional dropout, with the largest benefits on Flickr8k and Flickr30k, where over-fitting is likely.Conventional dropout produces lower output variance than tied dropout in the reported analysis.
- Ablation analysis: Leaky ReLU avoids a large correlation loss observed with conventional ReLU, while removing the hidden-layer loss term reduces correlation by 31%.On Flickr8k, conventional ReLU caused an approximately 33% correlation loss, and removing Lh produced a 31% decrease.
5. Conclusions
The method links paired samples from two sources with a tied 2-way architecture and is reported to outperform literature methods across correlation-analysis approaches. Its generic design also applies to computer-vision domains using two modalities, while several training refinements address variance and optimization challenges.
- The method significantly outperforms classical, modern, and deep correlation-analysis methods for linking paired samples from two sources.
- A tied 2-way architecture reconstructs between views in both directions while using the Euclidean loss.
- Variance-maintaining training modifications work together to produce a highly accurate architecture.The paper describes these contributions as including regularization and normalization changes aimed at maintaining learned-representation variance.
- The method is generic for computer-vision domains that use two data modalities and its variance-injection method can be incorporated into existing networks.
- Future work includes exploring tied 2-Way networks further, eliminating near-zero batch-normalization biases, and adding hidden layers progressively during training.