Omnivore: A Single Model for Many Visual Modalities

1. Introduction

• The introduction argues that modality-agnostic models could enable cross-modal learning, make labeled datasets more reusable, reduce modality-specific engineering, and accommodate new visual sensors.It frames such models as a first step toward vision systems that work across modalities, unlike systems optimized separately for each modality.Source: §1 · pp. 1–2
• OMNIVORE converts images, videos, and single-view 3D into patch embeddings processed by the same Transformer and shared model parameters, training on standard datasets without explicit cross-modal correspondences.Images use patches, videos use spatio-temporal tubes, and single-view 3D uses RGB and depth patches.Source: §1 · p. 2, Figure 2 · p. 2
• 85.6% top-1 accuracy on ImageNet-1K, 83.4% top-1 on Kinetics-400, and 67.4% top-1 accuracy on SUN RGBD; OMNIVORE performs at par with or better than same-size modality-specific models.It also sets new state-of-the-art results on EPIC-Kitchens-100, Something Something-v2, and single-view 3D classification and segmentation benchmarks.Source: §1 · p. 2
• OMNIVORE representations generalize across visual modalities without explicit cross-modal supervision, a capability that emerges from sharing parameters across modality-specific data.The introduction gives recognizing pumpkins in 3D after seeing labeled pumpkin videos as an example of cross-modal generalization.Source: §1 · pp. 1–2
• OMNIVORE performs at par with recent large transformers on ImageNet-1K and sets new state-of-the-art results on action-recognition and single-view 3D benchmarks.The cited action-recognition benchmarks are EPIC-Kitchens-100 and Something Something-v2; the 3D results cover classification and segmentation.Source: §1 · p. 2

ConvNet architectures in vision

• I3D [13] inflates 2D convolutions into 3D for video and 3D recognition, but inflation applies only to finetuning, unlike Omnivore’s joint pretraining across modalities.Inflated pretrained image networks help video and 3D models because those datasets are relatively small.Source: §2 · p. 2

Transformers in vision

• Vision Transformers including ViT [21], Swin [51], and MViT [24] span visual tasks and modalities, but prior studies train separate models; Omnivore trains one model across them.Perceiver [38] can model image, point cloud, audio, and video inputs.Source: §2 · p. 3

Multi-modal learning

• Unlike multimodal fusion methods that generally require corresponding modalities available simultaneously, Omnivore trains across visual modalities without that assumption, using standard single-modality datasets.Its shared encoder naturally supports cross-modal generalization.Source: §2 · p. 3

Multi-task learning

• Multi-task learners predict multiple tasks from one input modality, whereas Omnivore performs one classification task across modalities; other multimodal learners use hand-designed components for each modality.Multi-task learning is known to work well when target tasks exhibit strong similarities [61] [99].Source: §2 · p. 3

3. Approach

• Transformer self-attention handles variable-sized visual inputs, addressing differences such as video’s temporal axis and 3D data’s depth channel [92].The model is designed to operate on images, videos, and single-view 3D data.Source: §3 · p. 3

3.1. The OMNIVORE Model

• Images, videos, and single-view 3D inputs become 4D tensors, with RGB images and 3D inputs represented as single frames and videos as multiple frames.Single-view 3D inputs have three RGB channels and one depth channel.Source: §3.1 · p. 3
• Non-overlapping patches are mapped into a shared representation using shared RGB embedding layers and a separate depth-channel embedding added to its corresponding RGB patch.Image and 3D RGB patches are zero-padded to match the video patch shape before embedding.Source: §3.1 · p. 3
• The shared patch embeddings are processed by a Swin transformer [51], using local spatio-temporal self-attention and separate spatial and temporal relative positional encodings.The architecture can use vision transformers generally, and Swin is selected as the base model.Source: §3.1 · pp. 3–4

3.2. Training the OMNIVORE Model

• Omnivore is trained with cross-entropy and mini-batch SGD on classification datasets, using dataset-specific classifiers without requiring aligned inputs or shared label spaces.Most models are trained jointly on ImageNet-1K, Kinetics-400, and SUN RGB-D; the paper evaluates batches drawn separately by dataset and batches mixing samples across datasets.Source: §3.2 · p. 4

4. Experiments

• Experiments evaluate joint pretraining on ImageNet-1K, Kinetics-400, and SUN RGB-D, then test transfer across image, video, and single-view 3D tasks.Image transfer includes iNaturalist-2018, Oxford-IIIT Pets, and Places-365; video transfer includes Something-Something-v2 and EPIC-Kitchens-100; 3D transfer uses NYU-v2. The Swin backbone is trained with dataset-specific heads and RGB-channel dropout for SUN RGB-D.Source: §4 · p. 4

4.1. Comparison with Modality-Specific Models

• Across model sizes, OMNIVORE matches or exceeds modality-specific pretraining baselines with the same training data, parameter count, and model capacity.The larger Swin-B improves over modality-specific models on IN1K and K400, unlike Swin-T; on K400, it particularly improves classes involving human body parts.Source: §4.1 · p. 5, Table 2 · p. 5, Figure 3 · p. 5
• After finetuning, OMNIVORE transfers better than modality-specific models on nearly all seven downstream tasks, with significant video gains despite no additional pretraining video supervision.The models have the same capacity as the baselines, and higher-capacity models benefit most from omnivorous training.Source: §4.1 · pp. 5–6, Table 3 · p. 5

4.2. Comparison with the state-of-the-art

• 84.1% accuracy on K400 gives OMNIVORE a 1% gain over prior state of the art, while it matches state-of-the-art performance on IN1K.The Swin-B and Swin-L models were pretrained on IN21K, IN1K, K400, and SUN; the comparison covers all three pretraining benchmarks.Source: §4.2 · p. 6, Table 4 · p. 6
• OMNIVORE achieves state-of-the-art results on Places-365, iNaturalist-2018, Oxford-IIIT Pets, EPIC-Kitchens-100, SSv2, and NYU-v2 classification and segmentation.On EPIC-Kitchens-100, Swin-B achieves the best performance across verb, noun, and verb-noun action classification; on NYU-v2, it operates directly on disparity using a generic architecture.Source: §4.2 · p. 6, Table 5 · p. 6, Table 6 · p. 7

4.3. Ablation Study

• Ablations find performance relatively stable across design choices, with scratch training improving image and video classification and the separate depth embedding improving SUN performance.Higher SUN replication hurts SUN performance; increasing IN21K's share decreases K400 performance but improves SUN, leading to the final 0.1:1:1:10 setting. The batching strategies perform similarly, and the separate depth layer also improves downstream RGBD transfer.Source: §4.3 · pp. 6–7

Cross-Modal Generalization

• Omnivore retrieves semantically similar depth maps for ImageNet-1K RGB queries despite never training on ImageNet-1K depth maps or predicted depth.The authors attribute this ability to an almost entirely shared encoder, not explicitly learned cross-modal correspondences.Source: §5 · p. 7, Figure 4 · p. 8
• Omnivore (Swin-B) reports 84.3 for RGB, 63.1 for D, and 83.7 for RGBD in cross-modal classification experiments.The experiments use k-NN with k = 20, querying ImageNet-1K validation RGB representations against training-set images, RGBD images, and depth-only images.Source: §5 · p. 8
• RGB queries also retrieve related depth maps and Kinetics-400 videos, demonstrating retrieval across RGB images, RGBD data, and RGB videos in a shared representation space.The paper further reports successful k-NN classification when retrieving depth images, despite their providing less object-class information than RGB images.Source: §5 · p. 8
• At 1-frame inference, Omnivore outperforms VideoSwin-B by 18.5% and degrades more gracefully as clip length decreases.The model is trained on 32-frame clips; at inference, clip lengths vary from 1 to 32 while clips still cover the full 10-second video.Source: Figure 5 · p. 8, §5 · p. 8

Limitations

• Omnivore supports only single-view 3D images, and averaging predictions across rendered views of other representations would not leverage multi-view information effectively.The implementation does not generalize to voxels or point clouds.Source: §6 · p. 8
• Depth inputs are not scale-invariant, so the authors used normalizations to alleviate this issue.Source: §6 · p. 8
• Omnivore uses only visual modalities and classification pretraining; audio and structured prediction tasks such as segmentation are left for future work.The authors suggest structured prediction might yield richer representations.Source: §6 · p. 8

Ethical Considerations

• Real-world deployment should be preceded by analysis of ethical problems, including performance disparities, harmful associations, and predictions that may propagate stereotypes.Source: §6 · p. 8

A. Implementation details for Pretraining

• Pretraining uses a shared 500-epoch AdamW recipe with batch size 4096 per dataset, cosine scheduling, EMA, and stochastic depth.The schedule uses linear warm-up and cool-down during the first and last 10% of training, respectively, with a peak LR of 2 · 10−3 and weight decay of 5 · 10−2. Swin-T/S/L use window size 8×7×7 and Swin-B uses 16×7×7; stochastic-depth drop rates are 0.1 for Swin-T, 0.2 for Swin-S, and 0.3 for Swin-B/L. EMA uses decay 10−4, and the best training results are reported because EMA results peak before training ends.Source: A. Implementation details for… · p. 12
• Augmentation and regularization vary by dataset, with ImageNet, SUN RGB-D, and Kinetics-400 using distinct configurations.IN1K and IN21K use RandAugment [19], mixup [101], CutMix [98], label smoothing [85], Random Erasing with settings from [88], and color jittering of 0.4. SUN RGB-D disparity is clamped and normalized, RGB is dropped with probability 0.5, and pretraining with ImageNet-21K adds 0.5 Dropout before the linear head. Kinetics-400 uses mixup, CutMix, label smoothing, and 0.5 Dropout before the linear head.Source: A. Implementation details for… · p. 12

B.1. Image Classification

• Image models are finetuned for 100 epochs with SGD, strong augmentations, and grid-searched learning rates and drop path.Evaluation uses a 224 × 224 center crop; 384px evaluation uses a 384 × 384 center crop and increases Swin spatial window size from 7 to 12.Source: §B.1 · p. 12

B.2. Video Classification

• For Something-Something-v2, video models are finetuned for 60 epochs with AdamW, warmup and cosine decay, and a backbone learning rate 0.1× the head rate.The learning rate warms from 10^-6 to 6·10^-3 over 5% of training, then decays to 10^-6.Source: §B.2 · p. 12
• Omnivore improves per-class F1-score across all 38 K400 action groups versus an ImageNet-1K-pretrained baseline fine-tuned on K400.Omnivore was jointly trained on ImageNet-1K, K400, and SUN RGB-D.Source: Figure 6 · p. 12
• For EPIC-Kitchens-100, finetuning uses a peak learning rate of 2·10^-3 for 150 epochs, with weight decay 0.05 and batch size 4×64 across 64 GPUs.The settings were used for both the modality-specific baseline and Omnivore.Source: §B.2 · p. 12
• Video evaluation samples 32-frame clips at stride 2 and averages predictions over three 224×224 spatial crops.Training uses a 224px random resized crop after scaling the short side to 256px, followed by RandAugment and Random Erasing.Source: §B.2 · p. 12
• For IN21K-pretrained Omnivore state-of-the-art comparisons, Something-Something-v2 uses a 1.2·10^-3 peak learning rate for 150 epochs, while EPIC-Kitchens-100 uses 4·10^-4 for 100 epochs.The EPIC setting uses weight decay 0.004 and the same schedule for backbone and head; both tasks also use CutMix, label smoothing, and EMA.Source: §B.2 · p. 12

• NYU scene classification follows and uses 10 classes derived from the original 19.Swin B and Swin L models were trained for 200 epochs with a starting learning rate of 5 × 10−3; weight decay was 0 for Swin B and 1 × 10−4 for Swin L.Source: §B.3 · pp. 12–13
• OMNIVORE improves per-class F1 on 308 of K400’s 400 classes versus an ImageNet-1K-pretrained, K400-finetuned baseline.OMNIVORE was jointly trained on ImageNet-1K, K400, and SUN RGB-D.Source: Figure 7 · p. 13
• NYU RGB-D segmentation follows [10], using a Swin trunk with an UperNet [95] head and AdamW [53] fine-tuning.Training uses weight decay 0.01, polynomial learning-rate decay (power 1) starting at 0.00006, 1500 warmup iterations, and batch size 32; NYU depth maps are converted to disparity using the Kinect camera baseline and focal length.Source: §B.3 · p. 13

B.4. k-NN experiments

• The ImageNet-1K depth inputs were generated by applying pretrained DPT Large to the training set with the preprocessing steps specified in.Source: §B.4 · p. 13
• For k-NN evaluation following [12], stage 3 [51] features were L2-normalized; RGB, RGBD, or depth features came from training images, while validation queries used RGB.Source: §B.4 · p. 13
• Each of the top-k neighbors contributes its one-hot ground-truth label weighted by e^(s/τ), where s is feature similarity and τ is set to 0.07; the weighted labels are summed.Source: §B.4 · p. 13

C. Other Results

• OMNIVORE outperforms the modality-specific baseline when finetuned on low-shot versions of Places-365.The passage identifies these models as those from Table 3; Table 10 presents the low-shot finetuning results.Source: C. Other Results · p. 13, Table 10 · p. 14
• OMNIVORE pretraining is effective on UCF-101 for RGB-only sports action recognition, although state-of-the-art results often use extra features such as optical flow or dense trajectories.Source: C. Other Results · p. 13
• Per-class comparisons report OMNIVORE gains over the VideoSwin baseline in Figs. 6 and 7.Source: C. Other Results · p. 14