Source-linked AI summary
CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth
Jose M. Facil, Benjamin Ummenhofer, Huizhong Zhou, Luis Montesano, Thomas Brox, Javier Civera
TL;DR
Single-view depth networks often fail to generalize across camera models, making camera changes a barrier to reusing training data. The paper introduces CAM-Convs, which incorporate camera parameters into convolutions, and reports improved cross-camera generalization without reduced depth accuracy. The method can also exploit images from different cameras, although focal-length normalization has a constant-pixel-size assumption.
Problem
Single-view depth networks trained and evaluated with the same camera may not generalize when camera parameters differ between training and testing.
Method
CAM-Convs concatenate camera internal parameters with feature maps so depth networks learn calibration-dependent patterns.
Results
CAM-Convs generalize across camera models while maintaining depth prediction accuracy and outperforming standard networks in cross-camera settings.
Takeaways & Limitations
CAM-Convs allow depth networks to exploit training images from different cameras and generalize to cameras unseen during training.
Takeaways & Limitations
Focal-length normalization assumes constant pixel size across the image set, whereas CAM-Convs are presented as overcoming this limitation.
Abstract
from arXiv · showhide
Single-view depth estimation suffers from the problem that a network trained on images from one camera does not generalize to images taken with a different camera model. Thus, changing the camera model requires collecting an entirely new training dataset. In this work, we propose a new type of convolution that can take the camera parameters into account, thus allowing neural networks to learn calibration-aware patterns. Experiments confirm that this improves the generalization capabilities of depth prediction networks considerably, and clearly outperforms the state of the art when the train and test images are acquired with different cameras.
1. Introduction
Single-view depth benchmarks often use the same RGBD camera for training and testing, masking poor generalization across camera models. The paper introduces CAM-Convs to incorporate camera internal parameters and preserve depth accuracy across differing cameras.
- Same-camera benchmark evaluation does not reveal whether depth methods generalize to RGB images from another camera.
- State-of-the-art single-view depth networks do not generalize when test-camera parameters differ from training-camera parameters.
- Training with diverse cameras alone does not solve cross-camera generalization, and adapting current methods requires architectural changes.
- CAM-Convs address camera-internal-parameter variability, enabling training and testing with different cameras without performance degradation.
- CAM-Convs concatenate camera internal parameters with feature maps so networks learn depth dependence on calibration and generalize to unseen cameras.
2. Related Work
Prior single-view depth work spans supervised and unsupervised learning, multiple views, and diverse datasets, but camera parameters are generally not exploited. CAM-Convs explicitly incorporate internal camera parameters to improve generalization.
- Deep-learning research has addressed 3D structure and motion through supervised and unsupervised, single- and multi-view, and geometry-fused approaches.
- Single-view depth methods progressed from geometric and MRF formulations to multi-scale and later deep-learning architectures.
- Several methods adapt across datasets, but treat datasets as different tasks and require retraining.
- Methods using diverse internet images generally omit camera parameters, relying on dataset diversity for generalization.
- CAM-Convs explicitly implement internal camera parameters in the network to improve generalization.
3. Camera-Aware Multi-scale Convolutions
CAM-Convs inject camera-dependent coordinate and field-of-view maps into convolutional features, allowing depth networks to model calibration-dependent patterns. Focal-length normalization complements this design but assumes constant pixel size, whereas CAM-Convs do not.
- CAM-Convs add camera intrinsics to convolutions between encoder and decoder, including every level and skip connection while preserving pretrained encoders.
- The method precomputes coordinate and field-of-view maps from focal length, principal point, and sensor size, then concatenates them with input features.
- Centered-coordinate channels encode principal-point location and vary with sensor size, resolution, and pixel size before bilinear resizing and concatenation.
- Field-of-view maps encode captured context and focal length, responding to changes in focal length, sensor size, and principal point.
- 3.1. Focal Length Normalization: Focal-length normalization predicts depth relative to a default focal length and denormalizes raw inverse-depth predictions.
- 3.1. Focal Length Normalization: Focal-length normalization accelerates convergence but assumes constant pixel size, a limitation that CAM-Convs overcome.
4. Model and Training
The model uses a DispNet-inspired encoder-decoder with skip connections, intermediate predictions, and multi-task outputs for inverse depth, confidence, and surface normals. Training combines depth, confidence, normal, and scale-invariant gradient losses with empirically chosen weights.
- Network Architecture: The network uses a DispNet-inspired encoder-decoder with skip connections between encoder and decoder feature maps.Its backbone is ResNet-50 pretrained on ImageNet.
- Network Architecture: Intermediate pyramid-resolution predictions encourage task-focused internal features and faster convergence.
- Network Outputs: The network predicts inverse depth, confidence, and surface normals, with normals omitted at the two highest resolutions because their ground truth is too noisy.Inverse depth is chosen for its linear relationship with pixel variations.
- Losses: Training combines depth, confidence, normal, and scale-invariant gradient losses.The gradient loss is designed to favor smooth, edge-preserving depth estimates and uses five spacings.
- Losses: The total loss weights are λ1=150, λ2=100, λ3=50, and λ4=25.These weights are obtained empirically.
5. Multi-Camera Experiments and Results
The experiments isolate how camera intrinsics affect single-view depth generalization and compare naïve adaptation strategies with CAM-Convs. Standard networks overfit to focal length, sensor size, and context, whereas CAM-Convs generalize across unseen camera parameters and datasets.
- Experimental setup: The study varies focal length and sensor size while using 2D-3D Semantics data to isolate camera-parameter effects from content-distribution differences.The evaluation uses synthesized images with different intrinsics but the same content, plus 3-fold cross-validation and standard depth and inverse-depth metrics.
- Influence of context: Wider fields of view and larger sensor sizes provide more context, improving standard U-Net depth performance.The experiments compare focal lengths f128 and f64 at fixed sensor size, and sensor sizes s1 and s4 at fixed focal length.
- Overfitting of standard networks: Single-focal and single-sensor-size training overfit: performance drops when test cameras differ from the training camera, while multi-focal training with focal normalization helps but remains below same-camera baselines.Without focal normalization, performance degrades and training may converge poorly; focal normalization also does not model sensor-size and resolution changes.
- Overfitting of standard networks: Resizing generally degrades accuracy because it creates inconsistent training and test data, helping only when visual-feature distributions do not overlap.On KITTI and ScanNet, resizing reaches same-camera-baseline error because the datasets have completely different visual-feature distributions, a stated special case.
- Robust generalization with CAM-Convs: CAM-Convs outperform same-camera baselines on most test sets, including unseen sensor sizes and a camera unseen during training.The method learns dependencies between image features and calibration parameters, and the extreme unseen-camera case uses a larger sensor size and smaller focal length.
- Experiments on multiple datasets: Across NYUv2, KITTI, and MegaDepth, one CAM-Convs network produces stable predictions across camera models and data distributions without dataset- or camera-specific fine-tuning.CAM-Convs show lower error and variance across six cameras than the compared method, despite not being trained on NYUv2.
6. Conclusions
CAM-Convs address camera-related generalization in single-view depth prediction by incorporating camera intrinsics into the prediction process. This preserves accuracy on new cameras and improves use of training data from different cameras.
- CAM-Convs are a novel convolution type designed to make depth prediction networks camera-independent.They incorporate camera intrinsics jointly with image features when predicting depth.
- Current networks overfit to the training camera model, causing poor generalization to other cameras and degraded performance with mixed-camera training data.
- CAM-Convs maintain prediction accuracy for new cameras while better exploiting training data from different cameras.The paper identifies this as a direction for scaling systems that depend on camera parameters.