PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperbolic Image-Text Representations

Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, Ramakrishna Vedantam

arXiv:2304.09172v3cs.CVcs.LG

TL;DR

Existing vision-language models such as CLIP do not explicitly represent the hierarchy linking generic textual concepts to increasingly specific images. MERU addresses this gap with contrastive hyperbolic image-text representations, and the authors report structured, interpretable representations that are competitive with CLIP on standard multimodal tasks. The paper also reports benefits for small embedding dimensions, while noting limitations in transfer evaluation and Euclidean linear-probe performance.

  • Problem

    Current large-scale vision-language models do not explicitly capture the visual-semantic hierarchy in which generic textual concepts entail more specific images.

  • Method

    MERU trains contrastive image and text encoders whose embeddings are transferred to a Lorentz hyperboloid and structured with hyperbolic distance and entailment objectives.

  • Results

    MERU is competitive or more performant than Euclidean approaches such as CLIP on zero-shot retrieval and image classification while capturing hierarchical knowledge.

  • Takeaways & Limitations

    MERU provides an interpretable representation space for reasoning about semantic specificity and reports gains for small embedding dimensions useful in resource-constrained settings.

  • Takeaways & Limitations

    MERU’s image-encoder Euclidean representations underperform CLIP in linear-probe evaluations, while few-shot and full-model fine-tuning transfer remain outside the paper’s scope.

Abstract

from arXiv · show

Visual and linguistic concepts naturally organize themselves in a hierarchy, where a textual concept "dog" entails all images that contain dogs. Despite being intuitive, current large-scale vision and language models such as CLIP do not explicitly capture such hierarchy. We propose MERU, a contrastive model that yields hyperbolic representations of images and text. Hyperbolic spaces have suitable geometric properties to embed tree-like data, so MERU can better capture the underlying hierarchy in image-text datasets. Our results show that MERU learns a highly interpretable and structured representation space while being competitive with CLIP's performance on standard multi-modal tasks like image classification and image-text retrieval. Our code and models are available at https://www.github.com/facebookresearch/meru

1. Introduction

Images and text can be organized into a visual-semantic hierarchy, but conventional Euclidean representations do not naturally model its uneven neighborhood structure. MERU introduces hyperbolic image-text representations that capture this hierarchy and provide structured, interpretable embeddings while remaining competitive on multimodal tasks.

  • Visual-semantic hierarchy: Image descriptions vary in specificity, forming a hierarchy from generic captions such as “exhausted doggo” to detailed scene descriptions.The same image can support captions containing different amounts of semantic detail.
  • Visual-semantic hierarchy: Euclidean spaces apply the same distance behavior everywhere, making it difficult to place generic concepts near many related concepts while preserving specific relationships.Hyperbolic spaces address this through exponentially increasing volume away from the origin, allowing generic concepts near the origin and specific concepts farther away.
  • MERU: MERU learns hyperbolic image-text representations from image-text pairs, allowing the visual-semantic hierarchy to emerge without explicit hierarchical supervision.The model is designed as a large-scale contrastive approach inspired by existing vision-language models.
  • MERU: MERU is reported to improve image retrieval and classification, use the embedding space more efficiently, and support inference about relative semantic specificity.The authors also describe qualitative analysis for exploratory analysis of large-scale multimodal datasets.
  • Contributions: The method introduces deep hyperbolic representations trained with 12M image-text pairs and systematically compares them with a strong CLIP baseline.The comparison covers zero-shot retrieval, classification, and effectiveness at small embedding dimensions.

2. Preliminaries

Hyperbolic geometry models spaces with constant negative curvature, unlike flat Euclidean spaces, and can represent such spaces through models including the Lorentz hyperboloid. The preliminaries define the Lorentzian structure, geodesics, tangent-space operations, and exponential and logarithmic maps used by MERU.

  • Riemannian manifolds: A Riemannian manifold is a smooth manifold equipped with a smoothly varying inner-product metric on each tangent space.Euclidean space is a special case with the standard Euclidean inner product.
  • Hyperbolic geometry: Hyperbolic spaces are Riemannian manifolds with constant negative curvature, unlike flat Euclidean spaces with zero curvature.They cannot be represented in Euclidean coordinates while preserving both distances and angles.
  • Lorentz model: The Lorentz model represents an n-dimensional hyperbolic space as the upper half of a two-sheeted hyperboloid in R^(n+1).Its axes are described using time and space dimensions, borrowing terminology from special relativity.
  • Lorentz model: The Lorentzian inner product and curvature constraint define the hyperboloid as the set of vectors satisfying ⟨x,x⟩_L = −1/c for c > 0.This constraint specifies the constant-curvature manifold used in the Lorentz model.
  • Manifold operations: Geodesics are shortest paths on the manifold, while tangent spaces provide Euclidean vector spaces for local operations and projections.Ambient vectors can be orthogonally projected onto a tangent space, and exponential and logarithmic maps connect tangent-space vectors with manifold points.

3. Approach

MERU combines CLIP-like image and text encoders with Lorentz-hyperboloid embeddings and objectives that model semantic distance and text-to-image entailment. Its parameterization simplifies the exponential map while the combined losses enforce both alignment and hierarchy.

  • Model design: MERU processes images and text with separate encoders, then transfers their fixed-dimensional embeddings from Euclidean space to the Lorentz hyperboloid.The transfer uses a simplified exponential-map parameterization of space components, with the corresponding time component computed to keep embeddings on the hyperboloid.
  • Model design: Learnable image and text scalars rescale batch vectors before the exponential map to prevent numerical overflow.CLIP-style initialization can produce very large hyperbolic coordinates, such as approximately 6.7 × 10^10 for n = 512 and c = 1.
  • Entailment loss: An entailment loss induces a partial order by encouraging each paired image embedding to lie within the cone associated with its text embedding.The cone narrows farther from the origin, and the loss penalizes the exterior angle only when it exceeds the cone’s half-aperture.
  • Contrastive learning: MERU uses negative Lorentzian distance as the contrastive similarity measure instead of CLIP’s cosine similarity.The image-to-text and text-to-image losses average over positive and in-batch negative pairs, with logits divided by temperature τ.
  • Training objective: The total training objective is the minibatch average of contrastive loss plus λ times entailment loss.The entailment term is zero when the paired image already satisfies the cone-based partial-order relation.

4. Experiments

MERU is evaluated against CLIP on zero-shot retrieval and classification, including performance at smaller embedding widths and the effects of key design choices. Across these experiments, hyperbolic representations are generally competitive or superior, while specific objectives and distance formulations affect structure and training stability.

  • Experimental setup: MERU is compared with CLIP on zero-shot image-text retrieval, image classification, embedding widths, and ablations using large-scale image-text training.The experiments use ViT-S, ViT-B, and ViT-L encoders and transfer models without additional task-specific training.
  • Retrieval: MERU mostly performs best on COCO and Flickr30K retrieval, except Flickr30K text retrieval with ViT-B/16.The evaluation reports recall@5 and recall@10 and notes that increasing model size does not improve image retrieval for either model.
  • Classification: MERU matches or outperforms CLIP on 13 of 16 standard classification datasets.Both models have near-random performance on four additional benchmarks whose concepts are poorly covered in the RedCaps training data.
  • Embedding width: MERU consistently outperforms CLIP at lower embedding widths on zero-shot COCO retrieval and ImageNet classification.The models are evaluated with embedding widths from 64 to 512 dimensions, indicating an advantage when representation capacity is constrained.
  • Ablations: Removing the entailment loss has little effect on MERU’s quantitative performance but weakens structure and interpretability.The ablation supports the claim that hyperbolic geometry is sufficient for quantitative gains over CLIP, while entailment loss improves organization.
  • Ablations: Using the Lorentzian inner product directly in the contrastive loss makes large MERU models difficult to train because the loss diverges from numerical overflow.Lorentzian distance applies a logarithmic operator that slows growth and improves numerical stability.

5. Qualitative analysis

The qualitative analysis tests whether MERU organizes images and text into a visual-semantic hierarchy around a generic root concept. MERU separates text and image distances from the root and retrieves progressively more generic concepts along image-to-root traversals.

  • Root representation: The analysis treats images as leaf nodes, text descriptions as intermediate nodes, and [ROOT] as the most generic concept.For MERU, [ROOT] is the Lorentz hyperboloid origin; for CLIP, it is empirically estimated from the training embeddings.
  • Root distances: MERU places text embeddings closer to [ROOT] than image embeddings, whereas CLIP’s distance distributions overlap.MERU’s distributions resemble two concentric high-dimensional rings, with text more spread out and images relatively thin.
  • Image traversals: Image-to-root geodesic traversal retrieves textual concepts at progressively higher levels of abstraction.The procedure interpolates 50 equally spaced steps and retrieves nearest text embeddings at each step.
  • Image traversals: MERU retrieves a finer-grained hierarchy than CLIP, which often retrieves few or no intermediate captions between an image and [ROOT].The analysis uses selected Pexels images and captions, with additional results reported for other images and YFCC captions.

6. Related work

Prior vision-language models generally learn Euclidean representations, while related work studies entailment and hyperbolic embeddings in narrower settings. MERU combines multimodal contrastive learning with hyperbolic geometry and demonstrates zero-shot transfer at larger scale.

  • Vision-language representation learning: CLIP and ALIGN established scalable contrastive vision-language representations that support zero-shot recognition and retrieval, but use Euclidean embeddings.These models represent images and text as normalized vectors in a high-dimensional Euclidean space.
  • Entailment embeddings: Order Embeddings model the partial order between language and vision, while MERU combines multimodal structure with distance-based contrastive learning.The paper reports that distance-based contrastive learning was important for classification and retrieval performance.
  • Entailment embeddings: Related NLP and knowledge-graph methods embed partially ordered data or infer ordering from pairwise similarities.MERU shares aspects of both approaches because it imposes structure across modalities while allowing order to emerge within a modality.
  • Hyperbolic representations in computer vision: Earlier computer-vision work studied hyperbolic image embeddings, segmentation, and self-supervised representations, but not multimodal representations at MERU’s scale.The paper contrasts these efforts with MERU’s large-scale image-text training and zero-shot transfer across generic tasks.

7. Conclusion

MERU learns hyperbolic image-text representations that capture visual-semantic hierarchy while remaining competitive with Euclidean approaches. The paper also identifies noisy captions as a target for future data-quality improvements and reports important transfer limitations.

  • MERU captures hierarchical knowledge while remaining competitive or more performant than Euclidean representation approaches such as CLIP.This supports reasoning about images at different levels of abstraction.
  • Image traversals with MERU can help discover generic, noisy captions that create false negatives during contrastive training.Practitioners could filter or re-caption such images to improve dataset quality and train subsequent models.
  • MERU excels at zero-shot retrieval and image classification, but its image encoder’s underlying Euclidean representations underperform CLIP in linear-probe evaluations.The paper also leaves few-shot learning and full-model fine-tuning beyond scope.

A. Entailment loss derivations

The derivations define hyperbolic geometric quantities used by MERU’s entailment loss in the Lorentz hyperboloid model. They connect Poincaré-ball and Lorentzian formulations through isometry and simplify expressions using Lorentzian distances and hyperbolic identities.

  • For c > 0, the hyperboloid has curvature −c, which parameterizes the geometry used in the entailment-loss derivation.
  • The appendix derives the entailment-loss components used in MERU from these half-aperture and exterior-angle expressions.
  • Half-aperture: The half-aperture defines the entailment cone for a Poincaré-ball point and transfers to the Lorentz hyperboloid through an isometric transformation.The cone half-aperture remains invariant across the two hyperbolic models.
  • Exterior angle: The exterior-angle derivation considers the origin, text embedding, and image embedding as vertices of a hyperbolic triangle.It uses Lorentzian distances and the hyperbolic law of cosines before substituting auxiliary functions and identities.

B. Developing a strong CLIP baseline

The authors develop an accessible, strong CLIP baseline and use it to compare MERU fairly across zero-shot classification experiments. RedCaps training substantially strengthens the baseline relative to YFCC-trained alternatives.

  • The baseline is designed to train the smallest model on a single 8-GPU machine in less than one day.The original CLIP required a private 400M-pair dataset and more than 10 days across 128 GPUs.
  • 32.6% average accuracy is obtained by the publicly released SLIP ViT-S/16 checkpoint across the evaluation datasets.
  • 34.1% average performance is achieved by the authors’ re-implementation after three implementation changes.These changes concern cross-GPU gradient flow, weight decay, and image-resize interpolation.
  • 31.1% →37.1% average accuracy is obtained when training with RedCaps rather than YFCC.The improvement is especially pronounced for concepts well covered in RedCaps, including Food-101 and Pets.
  • 38.1% average performance is achieved by the final CLIP ViT-S/16 baseline across 16 datasets while training on 8 V100 GPUs in approximately 14 hours.

C. Linear probe evaluation

Linear-probe evaluation trains classifiers on frozen image-encoder embeddings rather than projected hyperbolic representations. MERU mostly matches or underperforms CLIP in this protocol, which is outside the paper’s primary zero-shot-transfer focus.

  • Linear probing trains logistic-regression classifiers on frozen image-encoder embeddings before the projection layer.For MERU, these underlying embeddings remain Euclidean rather than being lifted onto the hyperboloid.
  • The evaluation uses per-dataset regularization search for logistic regression before training final classifiers on combined training and validation splits.
  • MERU mostly matches or underperforms CLIP in linear-probe results.The paper focuses instead on zero-shot transfer and interpretability benefits.

D. Image traversals: more details and results

The traversal analysis probes MERU’s learned visual-semantic hierarchy by moving image embeddings toward [ROOT] and retrieving captions along the path. Across image categories, MERU is reported to produce more systematic and fine-grained hierarchies than CLIP.

  • Method: Image traversals interpolate 50 equally spaced steps from an image embedding y to [ROOT], retrieving text embeddings at each step.The traversal uses shortest paths and displays only unique captions when consecutive steps retrieve the same caption.
  • Method: MERU and CLIP use different interpolation procedures because their hyperbolic and Euclidean geometries differ.CLIP interpolates normalized embeddings directly, whereas MERU interpolates in tangent space before lifting points onto the hyperboloid.
  • Method: MERU retrieval first restricts candidates to captions that entail the step embedding, then selects the candidate with the highest Lorentzian inner product.[ROOT] is treated as entailing every embedding.
  • Retrieval set: The retrieval set combines manually collected Pexels captions and tags with processed YFCC-15M descriptions, yielding approximately 8.7M captions.Pexels tags are filtered to nouns and adjectives and converted into captions; YFCC captions undergo text cleaning and a 20-token length filter.
  • Results: Across locations, flora and fauna, food and drinks, objects, and scenes, MERU captures a more systematic and fine-grained visual-semantic hierarchy than CLIP.The paper presents selected examples for 60 images across the main and supplementary traversal figures.
Loading 2304.09172v3…