Source-linked AI summary
Multimodal similarity-preserving hashing
Jonathan Masci, Michael M. Bronstein, Alexander A. Bronstein, Jürgen Schmidhuber
TL;DR
Cross-modal similarity is difficult to model consistently, and prior hashing methods are limited by linear projections, ignored intra-modal similarities, or relaxed optimization. The paper introduces a coupled siamese neural network that jointly learns intra- and inter-modal similarities in compact binary codes. Experiments on multimedia retrieval datasets report that the method outperforms competing approaches, with much smaller codes and efficient search.
Problem
Cross-modal similarity learning must create a consistent measure across modalities, while prior approaches can ignore intra-modal similarities or be limited to linear projections and relaxed optimization.
Method
A coupled siamese neural network jointly learns intra- and inter-modal similarity-preserving hash embeddings without restricting mappings to linear projections.
Results
The method outperforms other approaches across reported quality criteria, achieves near-perfect performance with 8 bits in the multilayer setting, and produces codes at least 10× smaller.
Takeaways & Limitations
Multimodal similarity can be represented in compact binary codes while supporting efficient search and mappings of arbitrary complexity.
Takeaways & Limitations
Prior relaxation-based methods can yield suboptimal solutions, larger hash sizes, or inferior fixed-length performance, and may fail to capture data structure with linear embeddings.
Abstract
from arXiv · showhide
We introduce an efficient computational framework for hashing data belonging to multiple modalities into a single representation space where they become mutually comparable. The proposed approach is based on a novel coupled siamese neural network architecture and allows unified treatment of intra- and inter-modality similarity learning. Unlike existing cross-modality similarity learning approaches, our hashing functions are not limited to binarized linear projections and can assume arbitrarily complex forms. We show experimentally that our method significantly outperforms state-of-the-art hashing approaches on multimedia retrieval tasks.
1 Introduction
Similarity learning is especially difficult across modalities because meaningful cross-modal measures are hard to construct. Prior hashing and multimodal approaches improve efficiency but remain limited by side-information requirements, linear projections, or costly optimization; this paper proposes a unified neural framework.
- Motivation: Cross-modal similarity is difficult because modality-specific similarity functions do not necessarily yield a consistent measure across modalities.Examples include matching medical images from CT and MRI or searching jointly across audio, video, and text.
- Prior work: Supervised similarity learning uses side information such as labels, similar or dissimilar pairs, triplets, or metric dissimilarity values.These forms of supervision support learning representations for unseen data through direct parametric computation or out-of-sample extension.
- Prior work: Hashing methods encode data as binary codes and use Hamming distance for efficient similarity computation, but standard approaches cannot rely on side information.Locality-sensitive hashing and spectral hashing primarily approximate trusted similarities such as the Jaccard index or cosine distance.
- Prior work: Existing multimodal methods include cross-modal SSH and multimodal kernel learning, but they face limitations including ignored intra-modal similarities, linear projections, continuous metrics, or expensive semidefinite programming.These constraints affect the richness of learned similarities, representation compactness, or scalability.
- Contributions: The proposed framework jointly learns intra- and inter-modal similarity with a coupled siamese neural network architecture.It avoids relaxation-based optimization and can accommodate missing modalities while extending beyond the limitations of prior approaches.
2 Background
Multimodal similarity-preserving hashing learns binary embeddings for different modalities in one Hamming space while preserving intra- and inter-modal relationships. The full problem extends cross-modal hashing by incorporating within-modality similarities, and prior relaxed linear methods can yield larger codes or worse fixed-length performance.
- Problem definition: Multimodal hashing maps data from different modalities into a shared m-dimensional binary Hamming space using separate embeddings ξ and η.The embeddings aim to place similar points close and dissimilar points far apart across intra- and inter-modal comparisons.
- Problem definition: The full multimodal objective preserves intra-modal similarities dX and dY alongside inter-modal similarity dXY.This contrasts with cross-modality hashing, which considers only dXY and ignores dX and dY.
- Problem definition: Binary side information is sampled as similar and dissimilar pairs within and across modalities.The formulation represents these samples through positive sets PX, PY, and PXY and corresponding negative sets.
- Optimization: The optimization minimizes an aggregate of false-positive and false-negative rates across intra- and inter-modal pair sets.The objective combines expected Hamming dissimilarities for positive pairs and subtracts corresponding expectations for negative pairs.
- Prior and proposed methods: CM-SSH uses one-by-one boosted linear embeddings, whereas the proposed approach learns multimodal mappings through a siamese neural-network framework.The neural implementation can use a smooth tanh approximation to the sign activation during learning.
- Limitations of prior methods: Relaxation and linearization can produce suboptimal solutions, increasing hash sizes or reducing performance for a fixed hash length.The same approximation is also limited to linear embeddings that may not capture data structure well.
3 Multimodal NN hashing
The framework uses coupled siamese neural networks to learn similarity-sensitive hashes for two modalities, combining intra-modal and cross-modal similarity in one jointly trained model. Its neural-network embeddings can be extended from simple binary linear mappings to arbitrarily complex nonlinear forms.
- Similarity-sensitive hashing: The siamese loss uses squared embedding distances for positives and a hinge loss that pulls negative pairs at least mX apart.For binary vectors, squared Euclidean distance is equivalent up to constants to Hamming distance; the hinge term also provides robustness to outliers.
- Coupled siamese architecture: Two modality-specific siamese networks are coupled through a cross-modal loss, jointly learning separate parameter sets for modalities X and Y.The embeddings are ξ and η, with parameters (P, a) and (Q, b), respectively.
- Similarity-sensitive hashing: Tanh activations encourage binary representations, and thresholding the learned outputs produces the final hashes.The tanh output bounds the maximum distance by 4m; enlarging the dissimilar-pair margin encourages sign-different codes.
- Coupled siamese architecture: The model combines intra-modal and inter-modal similarity learning, while intra-modal losses can regularize training when reliable cross-modal samples are scarce.The intra-modal terms LX and LY are described as regularization in the full multimodal setting.
- Training: The coupled model is trained by back-propagation and can be optimized jointly with conjugate gradient or stochastic gradient descent.Gradients are computed with respect to the embedding parameters for both intra-modal and inter-modal losses.
- Non-linear embeddings: The framework generalizes to multilayer networks that model highly nonlinear embeddings and support fully online learning with constant memory requirements.The authors state that any class of neural networks can increase embedding complexity, while parameters are updated after each input tuple.
4 Results
Across ShapeGoogle, NUS, and Wiki retrieval experiments, the coupled siamese neural-network methods achieved strong multimodal hashing performance, with deeper and fully multimodal configurations improving results. The method also remained robust when cross-modal correspondences were reduced and produced compact codes for efficient search.
- Experimental setup: The experiments evaluate retrieval with mean average precision and rank matches using Hamming distance over learned binary codes.The ShapeGoogle study uses BoF and SS-BoF representations; the Wiki experiment uses 32-bit hashes, with some Euclidean-embedding results marked as non-comparable reference values.
- ShapeGoogle: NN-based CM-NN and MM-NN dramatically outperform CM-SSH at fixed hash lengths on ShapeGoogle.MM-NN reaches almost perfect performance with 12 bits, while CM-SSH requires almost 100 bits for similar performance.
- ShapeGoogle: Two-layer neural networks provide near-perfect ShapeGoogle performance with 8-bit hashes, substantially exceeding single-layer architectures.The added layer enables a nonlinear hashing function with markedly better performance than the linear single-layer version.
- ShapeGoogle: MM-NN consistently outperforms CM-NN on ShapeGoogle cross-modal retrieval, which the authors attribute to intra-modal losses acting as regularization.With only a subset of cross-modal data, CM-NN shows a significant performance drop, whereas MM-NN remains practically unchanged.
- ShapeGoogle: Intra-modal regularization makes cross-modal performance less sensitive to parameter choices, while MM-NN achieves higher performance than CM-NN for similar margins.The parameter study varies hash lengths and settings for single-layer networks; the zero-regularization setting corresponds to CM-NN.
- NUS: On NUS, MM-NN outperforms the other evaluated approaches across all reported quality criteria.The experiment compares MM-NN, CM-NN, and CM-SSH using 64-bit hashes and includes text-based image retrieval and image annotation examples.
- Efficiency: The approach produces binary codes at least 10× smaller that can be searched very efficiently.This compactness is presented as an advantage of the multimodal hashing framework.
5 Conclusions
The paper presents multimodal similarity-preserving hashing with a coupled siamese neural network, supporting mappings of arbitrary complexity and exact optimization. Experiments on multimedia retrieval datasets report performance superior to state-of-the-art hashing approaches.
- The framework uses a coupled siamese neural network architecture for multimodal similarity-preserving hashing.
- Unlike existing cross-modal methods, the approach is not restricted to linear projections and can train mappings of arbitrary complexity.
- The training procedure solves the exact optimization problem without boosting-based approximations.
- The method avoids semidefinite programming and scales to very large numbers of dimensions and training samples.
- Experiments on standard multimedia retrieval datasets showed performance superior to state-of-the-art hashing approaches.