PaperlayerView exact revision on arXiv

Source-linked AI summary

Omnivore: A Single Model for Many Visual Modalities

Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, Ishan Misra

arXiv:2201.08377v2cs.CVcs.AIcs.IRcs.LG

TL;DR

Existing vision systems usually specialize in individual modalities, limiting their ability to work across images, videos, and 3D data. OMNIVORE addresses this gap with one shared Transformer model trained jointly across standard datasets, matching or exceeding modality-specific models and generalizing across modalities.

  • Problem

    Prior work studies visual modalities in isolation with separate architectures, leaving limited support for a unified model across images, videos, and single-view 3D data.

  • Method

    OMNIVORE converts images, videos, and single-view 3D inputs into patch embeddings processed by shared Transformer parameters and trains jointly on classification datasets without requiring cross-modal correspondences.

  • Results

    OMNIVORE performs at par with or better than modality-specific models across standard benchmarks and sets new state-of-the-art results on several video and single-view 3D tasks.

  • Takeaways & Limitations

    Shared visual representations can support cross-modal recognition and transfer without explicit cross-modal supervision or corresponding data.

  • Takeaways & Limitations

    The current implementation supports only single-view 3D images, is not scale-invariant for depth inputs, excludes audio, and was pretrained only on classification.

Abstract

from arXiv · show

Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data. Instead, in this paper, we propose a single model which excels at classifying images, videos, and single-view 3D data using exactly the same model parameters. Our 'Omnivore' model leverages the flexibility of transformer-based architectures and is trained jointly on classification tasks from different modalities. Omnivore is simple to train, uses off-the-shelf standard datasets, and performs at-par or better than modality-specific models of the same size. A single Omnivore model obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN RGB-D. After finetuning, our models outperform prior work on a variety of vision tasks and generalize across modalities. Omnivore's shared visual representation naturally enables cross-modal recognition without access to correspondences between modalities. We hope our results motivate researchers to model visual modalities together.

1. Introduction

Omnivore targets a unified vision system by replacing modality-specific models with one shared model for images, videos, and single-view 3D data. It jointly trains on standard classification datasets and achieves competitive recognition while supporting cross-modal generalization.

  • Modality-agnostic models can transfer knowledge across modalities and make existing labeled datasets more usable.For example, a model could recognize pumpkins in 3D images after seeing labeled videos of pumpkins.
  • Shared architectures can reduce modality-specific research and engineering effort while accommodating additional visual sensors.Image and video models historically required separate development and tuning despite similar architectural progress.
  • OMNIVORE processes images, videos, and single-view 3D data using the same shared model parameters.It converts each modality into spatio-temporal patch embeddings processed by the same Transformer.
  • OMNIVORE representations generalize across modalities without explicit cross-modal correspondences or consistency losses.The paper attributes this capability to parameter sharing across models for different modalities.
  • 85.6% top-1 accuracy on ImageNet-1K, 83.4% top-1 on Kinetics-400, and 67.4% top-1 accuracy on SUN RGBD are reported for one OMNIVORE model.
  • OMNIVORE performs at par with recent large transformers on ImageNet-1K and reaches new state-of-the-art results on several action-recognition and single-view 3D benchmarks.The cited benchmarks include EPIC-Kitchens-100, Something Something-v2, and single-view 3D classification and segmentation.

2. Related Work

Related work has developed modality-specific convolutional and Transformer architectures for images, videos, and 3D data. Vision Transformers such as ViT, Swin, and MViT have shown strong performance across these domains, but typically with modality-oriented adaptations.

  • ConvNet architectures in vision: ConvNets use 2D convolutions for images, 3D convolutions for 3D data, and combinations of 2D and 3D convolutions for videos.I3D introduced inflating 2D image convolutions into 3D convolutions for video models.
  • Transformers in vision: Transformers originally developed for NLP have been applied successfully to image, video, and 3D vision tasks.
  • Transformers in vision: ViT, Swin, and MViT perform competitively on benchmark tasks including image classification, detection, and video recognition.Swin and MViT require minimal changes for image or video recognition.

3. Approach

OMNIVORE converts heterogeneous visual inputs into a common patch-embedding format and processes them with a shared Transformer. Joint classification training uses modality-specific datasets and output heads without requiring aligned observations or shared labels.

  • Input representation: The model represents images, videos, and single-view 3D inputs as 4D tensors with temporal, spatial, and channel dimensions.Images use one frame and three channels, videos use multiple frames, and single-view 3D uses one frame with RGB plus depth.
  • Input patches: Images become non-overlapping image patches, videos become spatio-temporal patches, and single-view 3D inputs split RGB and depth channels into separate patches.
  • Model architecture: OMNIVORE maps patches from all modalities into a shared representation using the same model parameters.The implementation can use a vision Transformer architecture and uses Swin as its base model.
  • Model architecture: Self-attention performs spatio-temporal modeling across patch embeddings, with separate relative positional encodings for spatial and temporal dimensions.
  • Training: Training jointly uses classification datasets such as ImageNet-1K, Kinetics-400, and SUN RGB-D, with dataset-specific linear classifiers over the shared representation.
  • Training: The approach does not require correspondences between modalities or a shared label space.

4. Experiments

The experiments evaluate OMNIVORE against modality-specific and state-of-the-art models across image, video, and single-view 3D tasks. Across pretraining and transfer settings, the shared model generally matches or outperforms specialized alternatives, with stronger gains at larger capacity.

  • Experimental setup: OMNIVORE is evaluated on standard image, video, and single-view 3D datasets, including ImageNet-1K, Kinetics, SUN RGB-D, and diverse transfer benchmarks.Transfer tasks cover fine-grained and scene image recognition, temporal video recognition, and RGBD scene classification and segmentation.
  • Comparison with modality-specific models: The comparison uses jointly trained OMNIVORE models and modality-specific ImageSwin, VideoSwin, and DepthSwin baselines with matched Swin architectures and parameter counts.Experiments include Swin-T, Swin-S, and Swin-B model sizes.
  • Comparison with modality-specific models: Across model sizes, OMNIVORE matches or exceeds modality-specific pretraining performance while using one shared model and one-third the parameters for deployment across modalities.The reported comparison uses the same training data, model parameters, and model capacity for the representations being evaluated.
  • Comparison with modality-specific models: Larger models benefit more from omnivorous training: Swin-B improves over modality-specific counterparts on both IN1K and K400, unlike Swin-T.The capacity-dependent pattern is reported for pretraining comparisons.
  • Transfer learning performance: OMNIVORE improves transfer performance on nearly all downstream tasks, with significant video-recognition gains despite no additional video pretraining supervision.The authors attribute the stronger out-of-distribution generalization to jointly pretraining on more diverse data, with larger models benefiting most.
  • Comparison with the state-of-the-art: Against state-of-the-art methods, OMNIVORE matches IN1K performance and reaches 84.1% accuracy on K400, a 1% gain over prior state of the art.The comparison uses IN21K alongside IN1K, K400, and SUN during pretraining for Swin-B and Swin-L models.
  • Comparison with the state-of-the-art: After finetuning, OMNIVORE outperforms prior state of the art on Places-365, iNaturalist-2018, and Oxford-IIIT Pets image classification.These results cover scene classification and fine-grained classification tasks.
  • Comparison with the state-of-the-art: OMNIVORE achieves the best reported EPIC-Kitchens-100 verb, noun, and action results and outperforms prior work on Something-Something-v2 temporal reasoning.The model sets a new state of the art while outperforming architectures specialized for video tasks.

5. Cross-Modal Generalization

OMNIVORE’s shared representation supports recognition and retrieval across RGB, depth, RGBD, and video inputs without explicit cross-modal correspondence training. It also remains effective under frame-level video inference and shows robust performance across design choices.

  • Ablation study: OMNIVORE’s ablation performance is robust to different training decisions, and scratch training slightly outperforms joint finetuning.The baseline ablation uses a 0.1:1:1:50 data ratio, separate batching, linear RGB/depth embedding layers, and 300 epochs.
  • Retrieval across images and depth: OMNIVORE uses shared representations to retrieve semantically similar depth maps from RGB queries without training on ImageNet depth maps.The retrieval database contains depth maps generated from the ImageNet-1K training set, while queries come from the validation set.
  • Classifying based on different modalities: 84.3 RGB, 63.1 D, and 83.7 RGBD are reported for OMNIVORE (Swin-B) in the modality-classification comparison.The passage reports the three values in RGB, depth, and RGBD order.
  • Retrieval across images and depth: OMNIVORE representations support k-NN classification across RGB, RGBD, and depth-only images using RGB-image representations.The experiment uses k = 20 on ImageNet-1K and retrieves examples from the training set.
  • Video clip-length robustness: OMNIVORE’s accuracy degrades more gracefully than VideoSwin-B as evaluated clip length decreases and remains effective at clip length 1.Both models are trained on 32-frame clips, while evaluation varies clip length using the same frame-sampling rate.
  • Video clip-length robustness: 18.5% is OMNIVORE’s advantage over the baseline at clip length 1 frame during full-video evaluation.The experiment varies clip length from 1 to 32 while proportionally increasing the number of clips to cover the full 10-second video.

6. Discussion and Limitations

OMNIVORE’s current scope is limited to single-view 3D images and visual inputs, with depth scale sensitivity requiring normalization. The models are pretrained only with classification, leaving structured prediction as a possible extension.

  • Scope boundaries: Current OMNIVORE supports single-view 3D images but does not generalize to voxels, point clouds, or other 3D representations.Rendering multiple views and averaging predictions would not effectively use multi-view information.
  • Input assumptions: Depth inputs are not scale-invariant, so OMNIVORE uses normalization to alleviate this issue.
  • Input assumptions: OMNIVORE uses visual modalities only and does not incorporate co-occurring modalities such as audio.
  • Training scope: The model is pretrained only with classification, while structured prediction tasks such as segmentation may yield richer representations.The paper leaves this extension to future work.

A. Implementation details for Pretraining

Pretraining uses AdamW, large per-dataset batches, cosine learning-rate scheduling, and modality-specific augmentation choices across ImageNet, SUN RGB-D, and Kinetics-400.

  • Optimization: Pretraining uses AdamW with batch size 4096 for each dataset and trains for 500 epochs.The schedule includes linear warmup and cooldown during the first and last 10% of training.
  • Optimization: The pretraining schedule uses peak learning rate 2 · 10^-3 and weight decay 5 · 10^-2.The learning rate follows a cosine schedule with warmup and cooldown phases.
  • Architecture: Swin-T, Swin-S, and Swin-L use window size 8×7×7, while Swin-B uses 16×7×7.
  • Augmentation and preprocessing: ImageNet pretraining uses RandAugment, mixup, CutMix, label smoothing, Random Erasing, and color jittering of 0.4.
  • Augmentation and preprocessing: SUN RGB-D preprocessing clamps and normalizes disparity, drops RGB channels with probability 0.5, and applies 0.5 Dropout before the linear head in ImageNet-21K pretraining.
  • Augmentation and preprocessing: Kinetics-400 pretraining uses mixup, CutMix, label smoothing, and Dropout of 0.5 before the linear head.

B. Details on the Transfer Tasks

Transfer-task evaluation combines SGD-based finetuning, task-specific AdamW schedules, standardized video preprocessing, and comparisons of action-group gains against a modality-specific baseline.

  • Finetuning setup: Finetuning uses 100 epochs of mini-batch SGD, half-wave cosine learning rates, zero weight decay, and grid searches over learning rate and drop path.The learning-rate grid spans 5e-3 through 6e-1, while drop path values are 0.1 and 0.3.
  • Video transfer tasks: Something Something-v2 finetuning uses AdamW for 60 epochs with warmup from 10^-6 to 6·10^-3 over the first 5% of training.The backbone uses 0.1× the classification-head learning rate.
  • Action recognition analysis: OMNIVORE improves per-class F1-score over the baseline for all 38 K400 action groups.The baseline is ImageNet-1K pretrained and K400 finetuned, whereas OMNIVORE is jointly trained on ImageNet-1K, K400, and SUN RGB-D.
  • Video transfer tasks: EPIC-Kitchens-100 evaluation uses a peak learning rate of 2·10^-3 and 150 training epochs.These settings are described as similar to Something Something-v2 except for the peak learning rate and training duration.
  • Video preprocessing: Video training samples 32 frames at stride 2, applies 224px crops, and averages predictions over three spatial crops at test time.Training uses temporal segment sampling, RandAugment, and Random Erasing.
  • State-of-the-art comparisons: For state-of-the-art comparisons, Something Something-v2 uses peak learning rate 1.2 · 10^-3 for 150 epochs, while EPIC-Kitchens-100 uses weight decay 0.004 for 100 epochs.The EPIC-Kitchens-100 setting also uses peak learning rate 4 · 10^-4 with the same schedule for backbone and head.

B.3. Single-view 3D Tasks

The paper reports single-view 3D scene classification and segmentation experiments alongside per-class action-recognition analysis. These results use established Swin-based setups and compare OMNIVORE’s gains against modality-specific baselines.

  • Per-class gains: 308 of 400 Kinetics-400 classes show improved F1 scores with OMNIVORE over the baseline.The baseline is ImageNet-1K pretrained and Kinetics-400 fine-tuned, while OMNIVORE is jointly trained on ImageNet-1K, Kinetics-400, and SUN RGB-D.
  • NYU RGBD Segmentation: NYU RGB-D segmentation follows the established training and evaluation setup with a Swin segmentation architecture and UperNet head.Models are fine-tuned with AdamW and weight decay of 0.01.

B.4. k-NN experiments

The k-NN experiments evaluate cross-modal recognition by extracting depth representations and classifying RGB validation images using nearest neighbors from the training set.

  • Depth extraction: Depth maps for ImageNet-1K training images are generated with a pretrained DPT large monocular depth-prediction model.The preprocessing follows the procedure provided by the depth-prediction model.
  • k-NN protocol: The evaluation extracts RGB, RGBD, or depth features for training images and RGB features for validation queries under the ImageNet-1K k-NN protocol.Stage 3 features are L2-normalized before nearest-neighbor retrieval.
  • Prediction rule: Each retrieved neighbor contributes a ground-truth one-hot label weighted by e^s/τ, with τ set to 0.07.The effective prediction sums the top-k weighted one-hot vectors; the same processing supports Figure 1 and Figure 4 visualizations.

C. Other Results

Additional experiments show OMNIVORE’s transfer benefits in video recognition and low-data image classification, including broad per-class gains over a video baseline.

  • UCF-101: OMNIVORE pretraining improves sports-action recognition on UCF-101 using RGB input only.The reported comparison averages performance over the benchmark’s standard three splits, while many state-of-the-art systems use additional features.
  • Low-data regime fine-tuning: OMNIVORE outperforms the modality-specific baseline when fine-tuned on low-shot versions of Places-365.The experiment is reported in the low-shot finetuning results.
  • Per-class gains: Per-class gains are presented for OMNIVORE relative to the VideoSwin baseline.The gains are shown in Figures 6 and 7.
Loading 2201.08377v2…