Source-linked AI summary
ImageBind: One Embedding Space To Bind Them All
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra
TL;DR
ImageBind addresses the difficulty of learning one embedding across modalities when complete multimodal pairing is infeasible. It aligns each modality with images, and reports emergent zero-shot and compositional capabilities across modalities, while acknowledging that task-specialist performance can remain higher.
Problem
Complete paired data across all modality combinations is infeasible, while pairwise embeddings remain limited to the modalities used during training.
Method
ImageBind learns a shared embedding by aligning images with text, audio, depth, thermal, and IMU data from multiple naturally paired datasets.
Results
ImageBind shows strong emergent zero-shot classification and retrieval across modalities, with capabilities improving as the underlying image representation becomes stronger.
Takeaways & Limitations
ImageBind enables cross-modal retrieval, embedding arithmetic, audio-based detection, and audio-conditioned image generation without training each capability directly.
Takeaways & Limitations
Because ImageBind is trained without a specific downstream task, its performance can lag specialist models.
Abstract
from arXiv · showhide
We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.
FAIR, Meta AI
ImageBind’s joint embedding aligns six modalities to enable multimodal capabilities, including retrieval, semantic composition, and audio-to-image generation.
- Aligning six modalities in a common embedding space enables cross-modal retrieval, embedding arithmetic, and audio-to-image generation.Audio embeddings can be used with a pretrained DALLE-2 decoder designed for CLIP text embeddings.
- ImageBind demonstrates emergent alignment between modalities that are not observed together.
1. Introduction
Existing multimodal methods generally align only modality pairs, while acquiring all combinations of paired data is infeasible. ImageBind instead uses image-paired data to bind six modalities and transfer zero-shot capabilities across them.
- Acquiring all types and combinations of paired data for the same images is infeasible.
- Prior methods typically align one modality pair, limiting embeddings to the modalities used during training.A major obstacle is the lack of large multimodal datasets containing all modalities together.
- ImageBind learns one shared representation by aligning each modality with images rather than requiring all modalities to co-occur.It combines web-scale image-text data with naturally paired video-audio, image-depth, and related data.
- ImageBind produces emergent zero-shot classification and retrieval across audio, depth, thermal, and IMU, while supporting compositional multimodal tasks.The reported applications include cross-modal retrieval, embedding arithmetic, audio-based detection, and audio-conditioned image generation.
2. Related Work
Prior work established strong image-text and other pairwise multimodal representations, but largely focused on limited modality combinations. ImageBind extends this line toward zero-shot recognition across multiple modalities.
- Vision-language pretraining uses linguistic signals to learn visual features for zero-shot recognition and text-to-image retrieval.
- Earlier multimodal work mostly considers image and text, whereas ImageBind targets zero-shot recognition across multiple modalities.
3. Method
ImageBind aligns each modality with images using naturally paired data and contrastive learning, producing a shared space where unseen modality pairs can align without direct paired training.
- Binding modalities with images: ImageBind aligns each modality’s embedding with image embeddings using naturally paired datasets such as image-text, IMU-video, and image-depth.
- Binding modalities with images: The method uses contrastive learning on paired observations to bring corresponding modality embeddings closer while separating unrelated observations.
- Binding modalities with images: Training on image-modality pairs yields emergent alignment between unseen modality pairs without observing direct training data for those pairs.This enables zero-shot and cross-modal retrieval, including text-audio classification without paired audio-text samples.
- Implementation Details: Evaluation covers zero-shot classification and retrieval datasets for audio, depth, thermal, and IMU without task-specific training or paired text data for those modalities.
- Implementation Details: The method encodes audio as mel-spectrograms, depth and thermal as one-channel images, and IMU signals with a convolution followed by a Transformer.
- Implementation Details: The implementation uses separate modality encoders and modality-specific linear projection heads to produce normalized fixed-size embeddings.Image and text encoders can be initialized from pretrained CLIP or OpenCLIP models.
4. Experiments
Experiments show that IMAGEBIND supports strong emergent zero-shot recognition and retrieval across modalities, competitive few-shot classification, and compositional cross-modal applications. Its image-aligned joint space transfers supervision across modalities and supports reuse in detection and generation without retraining.
- Emergent zero-shot classification: IMAGEBIND achieves strong emergent zero-shot classification across image, text, audio, depth, thermal, and IMU modalities, comparing favorably with supervised specialist models.The evaluation associates text embeddings with non-image modalities without observing those modality pairs together during training.
- Comparison to prior work: IMAGEBIND significantly outperforms prior methods on zero-shot audio retrieval and achieves comparable audio classification performance to supervised AudioCLIP.On Clotho, it doubles AVFIC’s performance despite using no audio-text pairing during training.
- Comparison to prior work: Combining IMAGEBIND audio and video embeddings further boosts text-based retrieval performance over using either modality alone.The audio-only model performs favorably against prior methods using video, while combining modalities improves retrieval.
- Few-shot classification: IMAGEBIND gains ∼40% top-1 accuracy on ≤4-shot audio classification over AudioMAE and matches or outperforms the supervised model on ≥1-shot classification.Its emergent zero-shot performance also surpasses the supervised ≤2-shot performance.
- Few-shot classification: IMAGEBIND significantly outperforms MultiMAE across all few-shot depth-classification settings, indicating strong generalization from image alignment.MultiMAE is trained on images, depth, and semantic segmentation data, whereas IMAGEBIND uses image alignment to train its depth features.
- Analysis and Applications: Adding image and audio embeddings enables retrieval of images combining both modalities’ concepts, such as fruits and birds.The joint space also supports replacing CLIP-based text embeddings with audio embeddings for object detection and segmentation without training.
5. Ablation Study
The ablations show that representation and training choices materially affect emergent zero-shot classification, with stronger image features improving unseen-modality recognition. Fixed temperatures, linear projection heads, longer training, and modality-dependent batch sizes are effective choices.
- Scaling the Image Encoder: 7% and 4% gains for depth and audio classification, respectively, result from replacing ViT-B with the stronger ViT-H image encoder.Other modality encoder sizes are fixed, isolating the effect of visual representation strength.
- Training Loss and Architecture: Fixed contrastive-loss temperatures outperform learnable temperature for depth, audio, and IMU classification.Higher temperatures work better for depth, thermal, and IMU, while lower temperatures favor audio.
- Training Loss and Architecture: Linear projection heads outperform MLP heads for depth and audio embeddings.This differs from standard self-supervised methods such as SimCLR, where MLP heads improve performance.
- Evaluating Vision Models: ImageBind can evaluate pretrained vision models by fixing the image encoder and aligning text, audio, and depth encoders with image-paired data.This setup measures how visual features affect multimodal tasks.
6. Discussion and Limitations
ImageBind provides a practical image-alignment route to joint multimodal embeddings and supports retrieval, zero-shot tasks, composition, and model evaluation. The authors identify specialist-model performance, task adaptation, benchmarking, and real-world deployment as remaining boundaries.
- Discussion: ImageBind trains a joint embedding using only image alignment and measures emergent alignment through cross-modal retrieval and text-based zero-shot tasks.The method also supports compositional multimodal tasks and evaluation of pretrained vision models on non-vision tasks.
- Limitations: The embeddings lag specialist models because they are trained without a specific downstream task.The authors suggest adapting general-purpose embeddings for individual tasks, including structured prediction such as detection.
- Limitations: ImageBind is a research prototype that cannot be readily used for real-world applications.
- Future Directions: Future improvements include adding other alignment data and creating benchmarks for emergent multimodal abilities.The authors mention text-paired and cross-modal data, such as audio-IMU pairs, as possible additions.
A. Datasets and Metrics
The evaluation uses naturally paired datasets for training and separate modality-specific datasets for zero-shot assessment. Audio, depth, video, and IMU inputs are standardized through fixed-length sampling and modality-specific representations.
- Audio Datasets: AudioSet provides 10-second YouTube videos across 527 classes, with about 2M unlabeled training videos used for audio-video matching and an 18K-video test split for zero-shot evaluation.AudioSet evaluation uses top-1 accuracy.
- Audio Datasets: ESC-50 contains 2,000 five-second audio clips across 50 environmental-sound classes and is evaluated with five-fold average top-1 accuracy.A single fold is used for ablations.
- Depth Datasets: SUN Depth-only and NYU-v2 Depth-only are evaluation datasets using disparity-processed depth maps and scene-class names for zero-shot templates.SUN Depth-only has approximately 5K validation depth maps and NYU-v2 Depth-only has 794 validation maps.
- Data Representations: Thermal inputs use single-channel images, depth uses disparity, audio uses mel-spectrograms, and IMU uses a 6×T sensor-reading tensor.These representations match the natural or empirically selected form described for each modality.
- Input Sampling: Audio and video are sampled as fixed-length clips, while IMU clips are five seconds centered on narration-aligned timestamps.Audio is sampled at 16KHz and converted to 128-bin log mel-spectrograms.
B.2. Few-shot evaluation details
Few-shot and multimodal evaluations freeze encoders or combine normalized embeddings under specified protocols. The appendix documents shared text templates, fixed sample counts, retrieval by cosine distance, and implementation settings.
- Few-shot Protocol: Few-shot evaluations sample k∈{1, 2, 4, 8} examples per class and train only a linear classifier with frozen encoder parameters.Models and baselines use exactly the same sampled examples.
- Few-shot Protocol: Audio few-shot ESC training uses AdamW with learning rate 1.6 × 10^-3, weight decay 0.05, and 50 epochs.
- Query Templates: The same CLIP query templates are used for non-visual modalities because their supervision is semantic and textual.
- Retrieval: Cross-modal retrieval uses temperature-scaled embeddings and cosine distance to compute nearest neighbors.
- Implementation: The reported pretraining experiments use 32GB V100 or 40GB A100 GPUs.
- Modality Combination: Audio and video embeddings are combined by linear interpolation, with weights 0.95 for video and 0.05 for audio.These weights produced the best reported combination.
C.2. Ablation setup
The ablation setup used 16-epoch training with ViT-Base encoders, and found that an MLP projection head did not improve performance.
- Ablation experiments used ViT-Base for both vision and modality-specific encoders and trained for 16 epochs unless otherwise noted.
- The MLP projection head did not improve performance over the linear head in these experiments.
D. Additional Results
The additional results demonstrate practical cross-modal search using IMU recordings retrieved by textual queries. The retrieved samples match the semantics of the text queries.
- Text queries can retrieve matching IMU recordings without training on IMU-text pairs.The examples use 3-channel accelerometer and gyroscope recordings, with accompanying video, for applications such as healthcare and activity search.
E. Additional Ablations
The additional ablations compare regression and contrastive objectives, while the paper notes that joint embeddings remain limited by dataset coverage and may create unintended associations.
- 31.74% ZS SUN top-1 accuracy from contrastive learning exceeded 29.04% from joint regression and contrastive training.Regression alone achieved 25.17%, so adding it to the contrastive objective did not improve performance.
- Joint embeddings may create unintentional associations and are limited to concepts present in the datasets used for each modality.The thermal datasets cover outdoor street scenes, while the depth datasets cover indoor scenes.