PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageBind: One Embedding Space To Bind Them All

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra

arXiv:2305.05665v2cs.CVcs.AIcs.LGcs.MM

TL;DR

Existing pair-trained embeddings are limited to their training modality pairs, while learning a joint space by collecting all combinations of paired data is infeasible. ImageBind aligns modalities to images to create a shared embedding space, reporting strong emergent zero-shot performance across modalities that compares favorably with specialist supervised models.

  • Problem

    Existing pair-trained embeddings are limited to training modality pairs, while acquiring paired data across all modality combinations is infeasible.

    Source: §1 · p. 1

  • Method

    ImageBind learns a shared space by aligning each modality with image embeddings using multiple types of image-paired data, without requiring all modalities to co-occur.

    Source: §1 · p. 2, §3 · p. 2

  • Results

    ImageBind reports strong emergent zero-shot classification and retrieval across audio, depth, thermal, and IMU, with audio results matching or exceeding specialist models trained with direct audio-text supervision on benchmarks.

    Source: §1 · p. 2, §4.1 · p. 4

  • Takeaways & Limitations

    The joint embeddings offer a way to evaluate pretrained vision models on non-vision tasks and support compositional multimodal tasks.

    Source: §6 · p. 8

    Because the embeddings are trained without a specific downstream task, they lag the performance of specialist models.

    Source: §6 · p. 8

Abstract

from arXiv · show

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.

FAIR, Meta AI

ImageBind aligns six modalities in a common embedding space, enabling cross-modal retrieval, semantic composition by adding embeddings, and audio-to-image generation.

Cross-Modal Retrieval

  • Aligning six modalities in a common space enables retrieval across modalities not observed together, including audio, depth, and text.The figure presents this as emergent cross-modal alignment.Source: Figure 1 · p. 1

Composing Modalities

  • Adding embeddings from different modalities naturally composes their semantics.Source: Figure 1 · p. 1

Audio-to-Image Generation

  • ImageBind audio embeddings can drive audio-to-image generation with a pre-trained DALLE-2 decoder designed for CLIP text embeddings [61].Source: Figure 1 · p. 1

1. Introduction

ImageBind learns one shared embedding space by aligning modalities through images, avoiding the need for datasets containing every modality combination. The paper reports emergent zero-shot and few-shot capabilities, including cross-modal retrieval and composition.

Motivation

  • Earlier image-text [1] [31] [46] [83] and audio [3] [4] [70] methods were limited to trained modality pairs, while fully co-occurring multimodal data are scarce.Images can associate with sensory experiences, but collecting all types and combinations of paired data for the same images is infeasible.Source: §1 · p. 1

Approach

  • IMAGEBIND learns a shared space by aligning each modality to image embeddings, using web-scale image-text pairs and naturally paired data instead of requiring every modality combination.The naturally paired data include video-audio and image-depth examples.Source: §1 · p. 2
  • It can initialize from CLIP, implicitly align text with modalities such as audio and depth, and enable zero-shot recognition without explicit semantic or textual pairing.The paper presents this as a way to leverage rich image and text representations and apply the method across modalities and tasks with little training.Source: §1 · p. 2

Results

  • IMAGEBIND shows strong emergent zero-shot classification and retrieval across audio, depth, thermal, and IMU; stronger image representations improve these properties, and audio results match or outperform directly supervised specialists.The audio comparison covers ESC, Clotho, and AudioCaps.Source: §1 · p. 2
  • Its embeddings also outperform specialist supervised models in few-shot evaluation and support cross-modal retrieval, embedding arithmetic, audio-source detection in images, and image generation from audio.These compositional uses are illustrated in Figure 1.Source: §1 · p. 2

2. Related Work

Prior research developed effective image-language embeddings and explored multimodal alignment, while ImageBind uses images as weak supervision rather than requiring explicit paired data across all modalities.

Language Image Pre-training

  • Image-language supervision supports zero-shot, open-vocabulary recognition and text-to-image retrieval [14] [38], while CLIP, ALIGN [31], and Florence [83] learn joint embeddings contrastively.Language supervision has also been used to learn video representations [2], and large-scale images with noisy captions yield strong visual features [34].Source: §2 · p. 2
  • CoCa [82] adds captioning to contrastive training, Flamingo [1] handles interleaved images and text with few-shot performance, and LiT finds freezing image encoders works best.This prior line mostly considers image and text, whereas ImageBind extends zero-shot recognition to multiple modalities.Source: §2 · p. 2

Multi-Modal Learning

  • Prior multimodal methods use supervised [21] [42] or self-supervised training [3] [20] [70], adapt CLIP to video [15] [43] [45], or add audio for zero-shot classification [27].Nagrani et al. use weakly labeled paired video-audio and captions for retrieval and captioning; ImageBind instead uses images as weak supervision without requiring explicit paired data across all modalities.Source: §2 · p. 2

Feature Alignment

  • CLIP features support teacher-based alignment [44] [58] [75] and zero-shot tasks including detection [24], segmentation [41], and mesh animation; PointCLIP applies CLIP to projected depth views.Implicit bridging in shared multilingual latent spaces also enables translation between language pairs without paired training data [33] [40].Source: §2 · p. 2

3. Method

IMAGEBIND learns a shared embedding space by aligning each modality with images, using their natural pairings to enable emergent alignment between modalities that were never paired during training.

3.1. Preliminaries

  • Unlike pair-specific contrastive spaces, IMAGEBIND enables zero-shot classification of modalities without paired text training.Prior zero-shot approaches for other modalities required specifically training on paired text data.Source: §3.1 · p. 3

3.2. Binding modalities with images

  • IMAGEBIND aligns image embeddings with text, audio, depth, thermal, and IMU embeddings using their natural pairings to build one joint space.The pairings include large-scale web image–text data and self-supervised pairings of other modalities with images.Source: §3 · p. 2, §3.2 · p. 3
  • For each aligned image–modality pair, IMAGEBIND trains normalized encoders with a symmetric InfoNCE loss that pulls matches together against in-batch negatives.The loss uses a temperature parameter to control the softmax distribution.Source: §3.2 · p. 3
  • Training only on image–modality pairs yields emergent alignment between unseen modality pairs, enabling zero-shot and cross-modal retrieval without training for those tasks.IMAGEBIND achieves state-of-the-art zero-shot text–audio classification without observing paired audio–text samples.Source: §3.2 · p. 3

3.3. Implementation Details

  • Separate modality-specific encoders and projection heads produce normalized d-dimensional embeddings; Transformer-based encoders use modality-specific inputs and processing.Audio uses 2-second, 16-kHz samples converted to 128-bin mel spectrograms, while IMU uses 5-second clips projected with a 1D convolution before Transformer encoding.Source: §3.3 · pp. 3–4

4. Experiments

The experiments test ImageBind’s image-bridged embedding across six modalities, evaluating emergent zero-shot and few-shot recognition alongside cross-modal applications. The results show strong transfer to non-visual modalities and enable reuse of existing vision models with audio embeddings.

4. Experiments

  • Training uses naturally paired image-related data and large-scale pretrained image-text encoders to connect six modalities, while updating only the audio, depth, thermal, and IMU encoders.The image and text encoders remain frozen; videos are processed as two-frame images, and small SUN RGB-D and LLVIP datasets are replicated 50× for training.Source: §4 · p. 4

4.1. Emergent zero-shot classification

  • IMAGEBIND achieves strong emergent zero-shot performance across benchmarks, compares favorably with supervised specialists, and aligns non-visual modalities such as audio and IMU through image-associated text supervision.The evaluation includes text-prompt classification of modalities not paired with text during training; standard zero-shot image and video results match OpenCLIP.Source: §4.1 · p. 4, Table 2 · p. 5

4.2. Comparison to prior work

  • IMAGEBIND outperforms prior work on zero-shot audio-text retrieval, matches supervised AudioCLIP on ESC classification, and gains further retrieval performance by combining audio and video.On Clotho it achieves double AVFIC’s performance without audio-text training; text-to-video retrieval reaches 36.1% R@1, and audio-only retrieval compares favorably with prior video retrieval.Source: Table 3 · p. 5, §4.2 · p. 5, Table 4 · p. 5

4.3. Few-shot classification

  • IMAGEBIND’s audio features outperform self-supervised AudioMAE across few-shot settings, match or exceed supervised AudioMAE, and its depth features beat MultiMAE throughout.Audio gains are ∼40% in top-1 accuracy at ≤4 shots, and emergent zero-shot audio performance surpasses supervised performance at ≤2 shots.Source: Figure 3 · p. 6, §4.3 · p. 6

4.4. Analysis and Applications

  • Adding image and audio embeddings composes their semantics, enabling image retrieval that reflects concepts from both modalities.The example combines an image of fruit with birdsong to retrieve images of fruit-bearing trees with birds.Source: §4.4 · p. 6, Figure 4 · p. 6
  • Without retraining, IMAGEBIND audio embeddings repurpose text-based vision models for audio-prompted object detection and sound-conditioned image generation.The experiments replace Detic’s CLIP class embeddings and a pretrained DALLE-2 model’s prompt embeddings with audio embeddings.Source: §4.4 · p. 6, Figure 5 · p. 6

5. Ablation Study

The ablations examine how image-encoder strength and modality-specific training choices shape emergent zero-shot classification, and use ImageBind to evaluate pretrained vision models on multimodal tasks.

5.1. Scaling the Image Encoder

  • Using ViT-H instead of ViT-B image features gains 7% on depth and 4% on audio classification, with stronger visual features improving emergent zero-shot performance across modalities.The experiment fixes the other modality encoders while varying the image encoder size.Source: Figure 6 · p. 7, §5.1 · p. 7

5.2. Training Loss and Architecture

  • Stronger image augmentation helps depth classification on the small SUN RGB-D paired dataset, but strongly augmenting video drops ESC audio performance by 34%.Basic augmentation significantly improves audio classification.Source: §5.2 · p. 7, Table 5 · p. 8
  • Spatially aligned image-depth crops outperform unaligned crops, while RandErase boosts depth and temporal alignment plus frequency masking improve audio classification.Unaligned crops degrade SUN-D performance by more than 10%; frequency masking provides a small audio boost.Source: §5.2 · p. 7, Table 5 · p. 8
  • A smaller depth encoder improves performance on the limited paired dataset, whereas a larger audio encoder helps, particularly with a high-capacity image encoder.The table caption also reports that a stronger image encoder improves both audio and depth tasks.Source: §5.2 · p. 7, Table 6 · p. 8
  • Fixed contrastive-loss temperature beats learnable temperature, linear heads beat MLPs for depth and audio, and longer training consistently improves zero-shot classification.A higher temperature works better for depth, thermal, and IMU, while a lower temperature works better for audio.Source: §5.2 · p. 7, Table 5 · p. 8
  • The optimal contrastive batch size varies by modality: smaller batches suit image-depth training, while larger batches improve audio-video results.The paper attributes this difference to dataset size and diversity, with more positive and negative pairs for audio-video.Source: §5.2 · p. 8, Table 7 · p. 8
  • With fixed pretrained image encoders, DINO outperforms supervised DeiT for emergent zero-shot depth and audio classification, and performance is not correlated with ImageNet vision performance.The authors propose ImageBind as a tool for measuring vision models’ strength on multimodal applications.Source: §5.2 · p. 8, Table 8 · p. 8

6. Discussion and Limitations

ImageBind uses image-only alignment to create emergent alignment across modalities, enabling cross-modal retrieval, text-based zero-shot tasks, compositional applications, and evaluation of vision models on non-vision tasks. The discussion identifies avenues for improving the embeddings and notes that the model remains a research prototype rather than a ready-to-use real-world system.

Discussion and Limitations

  • Image-only alignment produces emergent cross-modal alignment measurable through retrieval and text-based zero-shot tasks, while enabling compositional applications across modalities.The authors also describe using ImageBind to evaluate pretrained vision models on non-vision tasks and upgrade Detic and DALLE-2 to use audio.Source: §6 · p. 8
  • Alignment could be strengthened with additional paired data, including modalities paired with text or with one another, such as audio and IMU.Source: §6 · p. 8
  • Because the embeddings are trained without a specific downstream task, they lag specialist models; task adaptation, including for structured prediction such as detection, needs further research.Source: §6 · p. 8
  • New benchmarks, including measures of emergent zero-shot abilities, could support further research and new applications.Source: §6 · p. 8
  • ImageBind is a research prototype and is not readily usable for real-world applications.Source: §6 · p. 8

A. Datasets and Metrics … B.1. Inference implementation details

The appendix specifies datasets, representations, and inference procedures for evaluating ImageBind across audio, depth, thermal, and IMU tasks. Evaluation uses zero-shot classification and text-to-audio retrieval, with modality-specific preprocessing and sampling.

A. Datasets and Metrics

  • Audioset, ESC-50, and VGGSound [8] support zero-shot audio classification measured by top-1 accuracy; Audioset’s 2M unbalanced training videos are used unlabeled for audio-video matching.Audioset evaluation uses its 18K-video test split; ESC-50 reports the five-fold average, with only the first fold used for ablations, and VGGSound uses 14,073 test clips.Source: A. Datasets and Metrics · p. 12
  • Clotho and AudioCaps evaluate text-to-audio retrieval using recall@K, with each Clotho caption treated as a separate query.Clotho has 2,893 development and 1,045 test clips, each with five descriptions; AudioCaps uses 796 test clips after excluding VGGSound overlaps.Source: A. Datasets and Metrics · p. 12
  • SUN RGB-D supplies paired RGB-depth training data, while SUN-D and NYU-D provide depth-only zero-shot evaluation after depth maps are converted to disparity.LLVIP [32] supplies paired RGB-thermal training data and a balanced validation set of person and background boxes for zero-shot classification.Source: A. Datasets and Metrics · p. 12
  • Ego4D [23] evaluates zero-shot IMU scenario classification on synchronized clips, assigning each test clip the video-level scenario label.The filtered dataset contains 7,485 single-scenario videos, yielding 510,142 training clips and 68,865 test clips; training uses video frames and corresponding IMU signals.Source: A. Datasets and Metrics · pp. 12–13

A.1. Data Representations

  • Videos use two-frame spatiotemporal patches, inflated visual-encoder weights, and aggregated features across multiple clips so image-text models can be applied directly to video.The patch dimensions are 2×16×16 (T×H×W).Source: §A.1 · p. 13
  • Thermal input is single-channel, disparity works best among tested single-view depth encodings, audio uses mel-spectrograms, and IMU is represented as a 6×T tensor.Depth encodings tested include absolute depth, inverse depth, disparity, and HHA.Source: §A.1 · p. 13

B. Evaluation details

  • Evaluation details specify clip-based inference: audio and video use uniformly sampled clips, while IMU uses five-second clips centered on narration-aligned timestamps.Audio is sampled at 16 kHz and converted to 128-bin log-mel spectrograms with a 25 ms Hamming window and 10 ms hop; each IMU clip is 6×2000, and clips are independent test samples.Source: §B.1 · p. 13, B. Evaluation details · p. 13

B.2. Few-shot evaluation details … C.1. Best setup

These sections specify few-shot and zero-shot evaluation procedures, describe qualitative cross-modal applications, and report selected pretraining and modality-combination settings. They also show that the stated contrastive-loss batch sizes did not increase with the number of modalities.

B.2. Few-shot evaluation details

  • Few-shot evaluations sampled k ∈{1, 2, 4, 8} examples per class on ESC and SUN, used the same samples for the model and baselines, and trained only a linear classifier over frozen encoders.ESC audio used AdamW at 1.6 × 10−3 learning rate with 0.05 weight decay for 50 epochs; SUN depth used 10−2 learning rate with no weight decay for 60 epochs.Source: §B.2 · p. 13

B.3. Zero-shot evaluation details

  • Zero-shot evaluations used CLIP’s default query templates, including for audio and depth because supervision was semantic and textual.Source: §B.3 · p. 13

B.4. Qualitative evaluation details

  • Cross-modal retrieval used temperature-scaled embeddings and cosine distance, while arithmetic ℓ2-normalized and summed embeddings weighted by 0.5 before nearest-neighbor retrieval.The examples included audio from ESC, images from IN1K and COCO, depth from SUN-D, text from AudioCaps, and image-audio combinations retrieved from IN1K.Source: §B.4 · p. 13
  • Zero-shot audio-to-image generation used temperature-scaled ImageBind audio embeddings as proxies for CLIP text embeddings in an in-house DALLE-2 implementation [61].Source: §B.4 · p. 13
  • Audio object detection used 400 ESC validation audio descriptors from an ImageBind ViT-B/32 encoder as classifier embeddings in CLIP-based Detic, with a 0.9 score threshold.Source: §B.4 · pp. 13–14

C. Pretraining details

  • Pretraining hyperparameters for models reported in Table 4 are detailed in Table 9, and experiments ran on 32GB V100 or 40GB A100 GPUs.Source: §C.1 · p. 14, Table 9 · p. 14

C.1. Best setup

  • 2K for (video, audio) and 512 for (image, depth), (image, thermal), and (video, IMU) were used without increasing batch size as modalities increased, below prior-work batch sizes of >32K [10].Experiments sampled a mini-batch of one pair of modalities at a time.Source: §C.1 · p. 14
  • A linear combination of video and audio embeddings performed best with weights of 0.95 for video and 0.05 for audio.Embeddings from both modalities were extracted per sample before combining them.Source: §C.1 · p. 14

C.2. Ablation setup

The ablation evaluations use ViT-Base vision and modality-specific encoders, and the reported MLP head did not improve performance.

C.2. Ablation setup

  • Ablation experiments use ViT-Base for both vision and modality-specific encoders and train for 16 epochs unless noted otherwise.Source: §C.2 · p. 14
  • For Table 5b, the MLP head did not improve performance in these experiments.Source: §C.2 · p. 14

D. Additional Results

A shared embedding space enables text-based search of IMU data without training on IMU–text pairs, supporting applications such as healthcare and activity search.

D. Additional Results

  • IMAGEBIND retrieves IMU recordings from textual queries without IMU–text training pairs, with Figure 7 examples matching the query.The retrieved recordings show three-channel accelerometer and gyroscope data; the paper notes healthcare and activity search as potential applications.Source: D. Additional Results · p. 14

E. Additional Ablations · F. Ethical considerations

The ablation found that adding ℓ2 regression did not improve ZS SUN accuracy over contrastive training alone. The ethical discussion flags unintended associations, inherited web-data biases, and dataset-limited concepts in non-image modalities.

E. Additional Ablations

  • For ZS SUN top-1 accuracy, ℓ2 regression scored 25.17% alone or 29.04% with contrastive, below contrastive alone at 31.74%.The modality-specific encoders were trained to align with a frozen image encoder.Source: E. Additional Ablations · p. 15

F. Ethical considerations

Related papers

Loading 2305.05665v2…