OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images

1 Introduction

• Existing OOD benchmarks either conflate individual nuisance changes, rely on synthetic corruptions with limited real-world generalization, or use controlled settings whose conclusions may not transfer.Cross-dataset testing is described as a coarse measure, and some real-world nuisances, such as shape and 3D pose changes, are difficult to simulate.Source: §1 · p. 2
• OOD-CV provides real-image OOD examples for 10 object categories, with class labels, bounding boxes, and 3D poses for classification, detection, and pose-estimation benchmarks.Its test set varies from training data in individual nuisances including shape, texture, 3D pose, context, or weather.Source: §1 · pp. 2–3
• Each OOD sample is selected to vary in one nuisance while other factors remain similar to training data, enabling study of individual nuisance effects.The collection and annotation effort exceeded 650 hours.Source: §1 · p. 3
• Nuisance effects differ in magnitude and by vision task, while strong augmentation has only marginal real-world OOD benefits and can sometimes reduce performance.The introduction also reports no significant OOD-robustness differences between convolutional and transformer architectures.Source: §1 · p. 3

Robustness benchmark on synthetic images

• Synthetic-corruption benchmarks test image-processing noise or altered textures, but cannot mimic real-world 3D nuisances such as novel shape or pose and may be limited to one task.The cited experiments also find that style transfer and strong augmentation do not help with shape and pose changes.Source: §2 · p. 3

Robustness benchmark on real world images

• Real-image benchmarks capture distribution shifts, but examples such as ImageNet-A do not isolate nuisance factors, while ImageNet-R evaluates selected shifts and tasks.OOD-CV is presented as complementary because it disentangles semantic nuisances and supports multiple vision tasks.Source: §2 · p. 4

Techniques for improving robustness

• Robustness techniques include data augmentation and architectural changes, but prior evidence for transformers outperforming CNNs contrasts with this paper’s experiments.The benchmark is intended to evaluate such techniques on realistic data by nuisance and task.Source: §2 · pp. 4–5
• On realistic OOD data, current robustness approaches have only marginal effects and can even reduce robustness.The paper frames this finding as a reason for further work on robustness.Source: §2 · p. 5

3.1 What are important nuisance factors?

• The benchmark defines five independently varying, human-annotatable nuisances: object shape, 3D pose, texture, surrounding context, and weather.This ontology is intended to support measuring each nuisance’s OOD effect individually.Source: §3.1 · p. 5

3.2 Collecting images

• Internet searches use object-and-attribute keywords; images with multiple nuisances are filtered, yielding 2,632 images with shape, texture, context, and weather shifts.Only 10 PASCAL3D+ categories were used because sufficient OOD samples could not be found for bottle and television; pose and shape splits also use non-overlapping training and test variations augmented with internet data.Source: §3.2 · p. 6
• OOD-CV contains 13,297 images: 10,665 from PASCAL3D+ and 2,632 collected and annotated from the internet.Three annotators also filtered PASCAL3D+ training images that were too similar to OOD test images.Source: §3.2 · p. 6

3.3 Data annotation

• The annotations cover object class, 2D bounding box, and 3D object pose to support multiple vision tasks.The annotation process uses a CAD model and keypoints to align its 6D pose to the object; other annotators verify annotations.Source: §3.3 · p. 7

4.1 Robustness to individual nuisances

• Image classification is most affected by weather, detection by context, and 3D pose estimation by shape and pose; nuisance effects differ across tasks.The authors report that classification performance drops by 16.0% under weather shifts and Faster-RCNN drops by 37% mAP@50 under context shifts; pose estimation is most affected by shape and pose.Source: Table 1 · p. 8, §4.1 · pp. 8–9

4.3 Effect of Model Architecture on Robustness

• CNNs and transformers show comparable OOD performance across tasks, with similar degradation despite transformers’ higher in-domain performance.The reported differences between architectures are within the margin of error.Source: Figure 4 · p. 9, §4.3 · pp. 11–12

4.2 Data Augmentation for Enhancing Robustness

• Across all three tasks, augmentation mostly improves robustness to appearance shifts but can reduce performance under shape and 3D-pose shifts.AugMix gives the largest improvement for classification and pose estimation, while strong style-transfer augmentation improves detection more than adversarial training.Source: Table 2 · p. 10, §4.2 · pp. 10–11
• MobileNetV3 has degradation similar to ResNet-50, but augmentation improves context-shift performance by 0.9% versus 3.9% for ResNet-50.The authors suggest limited-capacity efficient models may be harder to make robust with augmentation.Source: Table 3 · p. 11, §4.3 · p. 12

4.3 Effect of Model Architecture on Robustness

• RetinaNet performs better than Faster-RCNN on OOD-CV, but their OOD degradation is similar after accounting for improved IID performance.The authors characterize two-stage methods as higher-scoring, but not necessarily more robust.Source: Table 4 · p. 12, §4.3 · p. 12
• NeMo is more robust than Res50-Specific to shape and pose shifts, while their robustness to appearance shifts is comparable.The authors connect explicit 3D geometry representations with improved robustness to geometry-based nuisances.Source: Table 5 · p. 12, §4.3 · p. 13
• Combining NeMo’s explicit geometry representation with strong augmentation improves appearance-shift robustness while retaining and slightly improving geometry-shift robustness.The results suggest that improving robustness to geometry-based and appearance-based nuisances can be developed independently.Source: §4.3 · p. 13

4.4 OOD shifts in Multiple Nuisances

• Combined shifts in appearance and geometry nuisances amplify performance loss across all three tasks; pose plus context reduces classification performance by 24.5%.The individual pose and context shifts reduce classification performance by 11.4% and 6.6%, respectively.Source: §4.4 · p. 13

5 Conclusion

• OOD shifts remain a major challenge for current state-of-the-art vision models, and their effects differ across vision tasks.The authors suggest different robustness solutions may be needed for different tasks.Source: §5 · p. 13
• Combined appearance- and geometry-based OOD shifts amplify each other, causing greater performance decreases than individual shifts.Source: Table 6 · p. 14
• Strong data augmentation improves robustness to appearance-based nuisances, while explicit 3D object representations improve robustness to geometry-based nuisances.The authors suggest combining explicit 3D geometry representations with strong augmentation to address both nuisance types.Source: §5 · p. 14

A Implementation Details

• The appendix section covers implementations of the models and robustness-improvement techniques used in Section 4.Source: A Implementation Details · p. 19

A.1 Image Classification

• The classification experiments compare MobileNetV3-Large, ResNet-50, and Swin-T using the same hyperparameters.Training uses batch size 64 for 100 epochs, with 224-by-224 inputs and a step-decayed learning rate initialized at 1e-4.Source: §A.1 · p. 19
• The evaluated image-augmentation techniques are style transfer, AugMix, and adversarial training.Style transfer and AugMix use implementations from their original authors and a PyTorch-based implementation, respectively; adversarial training uses the official implementation.Source: §A.1 · p. 19

A.2 Object Detection

• Object detection experiments use Faster-RCNN and RetinaNet, with hyperparameters held constant except where varied for study.Experiments mainly use Detectron2; style transfer and adversarial training are evaluated, while AugMix is omitted because its image-mixing step is harder to implement for detection.Source: §A.2 · p. 19
• Detection models train for 18,000 iterations with batch size 16 and multi-scale inputs, while testing resizes images to a short edge of 800.The learning rate starts at 0.02 and is multiplied by 0.1 at 12,000 and 16,000 iterations.Source: §A.2 · p. 19
• For 3D pose estimation, the experiments evaluate Res50-Specific and NeMo models.Each category-specific model trains for 800 epochs with batch size 108 and learning rate 1e-3.Source: §A.3 · p. 20

• Weather-nuisance images number 0 for chair, diningtable, and sofa because such examples were difficult to find.Source: B Detailed statistics · p. 20
• Table 1 gives the number of images in each category for each defined individual nuisance.Source: Table 1 · p. 20

• Example images from ROBIN are shown, and the full dataset will be released.Source: C Example Images from ROBIN · p. 20
• Additional example images are shown on the next page.Source: C Example Images from ROBIN · p. 20

D Images filtered from the original PASCAL3D+ dataset

• Images too similar to ROBIN test images were removed from original PASCAL3D+ to keep the test set out of distribution.Source: D Images filtered from the original… · p. 20
• The authors provide all images removed from the original PASCAL3D+ dataset in an anonymous repository; Figure 2 shows examples filtered out of that dataset, which were attached with the supplementary.Source: D Images filtered from the original… · p. 20, Figure 2 · p. 22

E Example images with multiple nuisances

• Images from the internet search that had multiple nuisances were also removed; Figure 3 gives examples.Source: E Example images with multiple nuisances · p. 22

F The user interface of our annotation tools

• The section shows interfaces for bounding-box and 3D-pose annotation tools, taken and slightly modified from GitHub project 8 and PASCAL3D+ dataset 9.Source: F The user interface of our annotation… · p. 22