Source-linked AI summary
3D Common Corruptions and Data Augmentation
Oğuzhan Fatih Kar, Teresa Yeo, Andrei Atanov, Amir Zamir
TL;DR
Computer vision models must handle naturally occurring shifts such as blur, illumination changes, and occlusion, but common approaches often ignore scene geometry. This paper introduces geometry-aware 3D Common Corruptions and matching augmentations, finding that they expose robustness vulnerabilities and improve predictions under realistic corruptions. The approach is efficient and broadly applicable, though its fidelity is bounded by 3D-data quality.
Problem
Models face naturally occurring geometric and semantic distribution shifts, motivating systematic robustness testing beyond uniform 2D corruptions.
Method
The paper generates geometry-consistent 3D Common Corruptions and 3D data augmentations using scene information such as depth, with efficient on-the-fly processing.
Results
3DCC exposes model vulnerabilities under realistic corruptions, while the proposed augmentations improve robustness compared with baselines.
Takeaways & Limitations
3D information is useful for both benchmarking and training models against real-world distribution shifts, particularly those centered around scene geometry.
Takeaways & Limitations
3DCC fidelity is bounded by the quality of the available 3D data and is currently an imperfect approximation of real-world 3D corruptions.
Abstract
from arXiv · showhide
We introduce a set of image transformations that can be used as corruptions to evaluate the robustness of models as well as data augmentation mechanisms for training neural networks. The primary distinction of the proposed transformations is that, unlike existing approaches such as Common Corruptions, the geometry of the scene is incorporated in the transformations -- thus leading to corruptions that are more likely to occur in the real world. We also introduce a set of semantic corruptions (e.g. natural object occlusions). We show these transformations are `efficient' (can be computed on-the-fly), `extendable' (can be applied on most image datasets), expose vulnerability of existing models, and can effectively make models more robust when employed as `3D data augmentation' mechanisms. The evaluations on several tasks and datasets suggest incorporating 3D information into benchmarking and training opens up a promising direction for robustness research.
1. Introduction
3D Common Corruptions (3DCC) introduces geometry-aware distribution shifts to test robustness against realistic camera, weather, occlusion, depth-of-field, and lighting changes. The framework also provides efficient, extendable 3D augmentations that improve robustness against such corruptions.
- Motivation and contribution: 3DCC incorporates scene geometry into corruptions, unlike uniform 2D modifications, producing shifts more likely to occur in the real world.Examples include motion parallax, depth-dependent fog, geometry-aware lighting, and viewpoint-based occlusion.
- Motivation and contribution: 3DCC includes 20 corruptions covering camera motion, weather, occlusions, depth of field, and lighting.
- Evaluation: 3DCC exposes drastic performance reductions in robustness methods and correlates with corruptions generated through photorealistic synthesis.These findings support using 3DCC as a challenging testbed for real-world corruptions, especially geometry-dependent ones.
- 3D data augmentation: 3D data augmentations significantly improve robustness against realistic corruptions, including corruptions that 2D augmentations cannot address.
- Practical properties: The corruptions are programmatic, parameterized, efficient to compute on-the-fly, and applicable to standard datasets without 3D labels.
2. Related Work
Prior robustness benchmarks largely use synthetic 2D corruptions or manually captured data, while related approaches often target specific corruptions or remove them. 3DCC instead uses 3D information to generate realistic, scalable corruptions for benchmarking and augmentation.
- Robustness benchmarks: Common Corruptions generates synthetic 2D corruptions, whereas 3DCC modifies real images with 3D information to expose different failure modes.
- Robustness benchmarks: ObjectNet provides realistic corruptions but requires significant manual effort and is not extendable, unlike the scalable generation targeted by 3DCC.
- Robustness analysis: Prior robustness analyses examine methods such as augmentation and self-supervised training, and 3DCC is used to test these methods under challenging shifts.
- Improving robustness: Existing robustness improvements include corrupted-data augmentation, texture changes, image compositions, transformations, self-training, pre-training, architectural changes, and ensembling, but gains can be nonuniform.
- Photorealistic synthesis: Photorealistic synthesis usually targets individual corruptions, whereas 3DCC combines related 3D transformations for robustness evaluation and improvement.
- Image restoration: Image restoration removes corruptions, while this work generates corrupted data for benchmarking and augmentation to encourage model invariance.
- Adversarial corruptions: Because naturally occurring shifts dominate many real-world failures, this work focuses on corruptions likely to occur in the real world rather than adversarial perturbations.
3. Generating 3D Common Corruptions
3DCC defines 20 corruption types that use scene geometry, depth, or 3D meshes to model realistic changes in focus, motion, weather, viewpoints, semantics, lighting, video, and noise. The pipeline supports calibrated, extendable corruption generation on datasets with or without ground-truth 3D information.
- 3DCC contains 20 corruptions spanning depth of field, camera motion, lighting, video, weather, view changes, semantics, and noise.
- Depth-of-field corruptions layer scene regions and blur them according to the pinhole camera model, producing near-focus and far-focus images.
- Camera-motion corruptions transform an RGB image into a depth-based point cloud and render views along a defined camera trajectory.
- Fog uses atmospheric light and depth-dependent transmission, while view and semantic corruptions alter camera parameters, object occlusion, or object scale.
- The released pipeline calibrates corruption intensities against 2DCC using average SSIM and applies to standard datasets with predicted MiDaS depth.
4. 3D Data Augmentation
3DCC corruptions can also augment training data by incorporating scene geometry to build invariance to realistic corruptions. These augmentations are generated on-the-fly with modest computational overhead.
- 3D data augmentation applies geometry-aware depth-of-field, camera-motion, and lighting corruptions during training.
- The augmentations are generated efficiently on-the-fly using parallel implementations, with depth of field taking 0.87 seconds for 128 images on one V100 GPU.Applying 2D defocus blur takes 0.54 seconds on average under the stated setup.
- Selected augmentation components, such as lighting illuminations, can be precomputed to increase efficiency.
5. Experiments
The experiments evaluate whether 3DCC exposes vulnerabilities missed by 2DCC, resembles expensive realistic synthetic corruptions, transfers to datasets without 3D information, and improves robustness through 3D augmentation.
- The experiments test 3DCC for vulnerability discovery, similarity to realistic synthetic corruptions, applicability without 3D information, semantic tasks, and robustness augmentation.
5.1. Preliminaries
The study evaluates 3DCC across vision datasets and tasks, emphasizing surface-normal and depth-estimation robustness while also considering classification and segmentation settings. It compares multiple robustness mechanisms and uses predicted depth to extend 3DCC beyond datasets with 3D labels.
- 3DCC is evaluated on tasks including surface normals and depth estimation, with additional results for panoptic segmentation and object recognition.
- ImageNet and COCO examples use MiDaS depth predictions to generate near-focus, far-focus, and fog corruptions without ground-truth 3D information.
- Robustness comparisons include DeepAugment, style augmentation, adversarial training, Cross-Domain Ensembles, 2DCC augmentation, and 3D data augmentation.
5.2. 3D Common Corruptions Benchmark
The 3DCC benchmark exposes model vulnerabilities across geometric, visual, semantic, and recognition corruptions, while remaining correlated with realistic synthetic effects and applicable using predicted depth.
- 5.2.1 3DCC can expose vulnerabilities: Existing robustness mechanisms improve over baseline performance but remain far from clean-data performance under 3DCC.Diverse 2D augmentation only partially helps against 3D corruptions.
- 5.2.1 3DCC can expose vulnerabilities: Training with larger, more diverse data and DPT architecture notably improves robustness, although models remain sensitive to corruptions such as z-motion blur and shadow.Taskonomy models are especially susceptible to field-of-view, camera-roll, and pitch changes compared with Omnidata-trained models.
- 5.2.3 Similarity to realistic synthetic corruptions: 3DCC corruptions are visually similar to Adobe After Effects effects while being cheaper to generate, and their error patterns correlate most strongly for matching near- and far-focus effects.The benchmark uses photorealistic Adobe After Effects synthesis as a proxy because labeled real corrupted data are expensive and scarce.
- 5.2.4 Effectiveness of applying 3DCC to other datasets: Predicted depth effectively approximates ground-truth depth for applying 3DCC, with strong error correlation of 0.79 across three corruptions.This supports applying 3DCC to datasets without ground-truth depth, with performance expected to improve with better depth predictions.
- 5.2.5 3DCC evaluations on semantic tasks: Semantic occlusions reduce IoU as occlusion increases, while ImageNet-3DCC significantly degrades robust recognition models and exposes vulnerabilities not captured by ImageNet-2DCC.Omnidata-trained models are generally more robust to occlusion than Taskonomy-trained models, with class-specific trends.
5.3. 3D data augmentation to improve robustness
The proposed 3D data augmentations extend baseline training with geometry-aware transformations and are evaluated across synthetic, DSLR, and in-the-wild data. The resulting models show substantially more robust qualitative predictions.
- 5.3. 3D data augmentation to improve robustness: The proposed model adds 3D augmentations on top of an Omnidata-trained DPT with 2DCC augmentation, enabling direct comparison of geometry-aware training.A further variant incorporates cross-task consistency constraints.
- 5.3. 3D data augmentation to improve robustness: ImageNet-3DCC provides an additional object-recognition benchmark for testing robustness against geometry-dependent corruptions.Models are compared using mean corruption error relative to AlexNet, following ImageNet-2DCC evaluation.
- 5.3. 3D data augmentation to improve robustness: Qualitative evaluations on OASIS, Adobe After Effects data, DSLR images, and YouTube videos show significantly more robust predictions from the proposed models.The evaluated tasks include surface-normal prediction, with ground truth unavailable for some YouTube examples.
6. Conclusion and Limitations
The framework uses 3D Common Corruptions to expose vulnerabilities and improve robustness against real-world plausible distribution shifts. The authors identify scope boundaries involving data quality, coverage, evaluation breadth, benchmark balance, augmentation use cases, and task selection.
- Conclusion: 3D Common Corruptions provides a challenging benchmark that exposes model vulnerabilities under real-world plausible corruptions, while the proposed augmentation improves robustness over baselines.The authors present 3D corruptions as useful for both benchmarking and training.
- Limitations: 3D quality: 3DCC fidelity is bounded by the quality of available 3D data, although higher-resolution sensory data and better depth prediction may improve it.The current benchmark is described as an imperfect but useful approximation of real-world 3D corruptions.
- Limitations: Non-exhaustive set: The corruption and augmentation set is not exhaustive, serving instead as a starter set that can generate more domain-specific shifts with minimal manual effort.The framework is intended to support expansion beyond the initial set.
- Limitations: Large-scale evaluation: The study does not provide a comprehensive robustness analysis because its main goal was to show that 3DCC exposes vulnerabilities.The authors encourage researchers to test additional models against the corruptions.
- Limitations: Benchmark balance: The benchmark does not explicitly balance corruption types, such as noise and blur, and could benefit from weighting strategies for calibrating average performance.The authors identify benchmark balancing as an open improvement.
- Limitations: Use cases and tasks: The experiments focus on robustness and dense regression, leaving applications such as self-supervised learning and other tasks for future investigation.The framework may apply to classification and other semantic tasks, but those settings were not evaluated here.