PaperlayerView exact revision on arXiv

Source-linked AI summary

From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Rit Gangopadhyay, Alex Wong

arXiv:2608.27860v1cs.CVcs.AI

TL;DR

Perspective-trained vision foundation models fail on fisheye imagery because radial distortion shifts latent features away from those learned from perspective images. DEX learns lightweight feature-space corrections with self-supervised alignment, improving depth estimation and open-vocabulary segmentation across fisheye settings while preserving frozen backbones. Its calibration decoding is useful but distortion-coefficient recovery remains difficult, and performance is bounded by backbone quality.

  • Problem

    Perspective-trained foundation models produce erroneous fisheye outputs because radial distortion induces a systematic shift in local pixels and latent embeddings.

  • Method

    DEX inserts lightweight learnable Extenders into a frozen backbone and aligns synthetically distorted fisheye embeddings with perspective embeddings using convex combinations and self-supervised loss.

  • Results

    DEX consistently improves monocular depth estimation and open-vocabulary segmentation across indoor and outdoor fisheye datasets and multiple backbone architectures.

  • Takeaways & Limitations

    Feature-space adaptation can extend perspective-trained models to fisheye imagery without explicit test-time rectification or backbone modification.

  • Takeaways & Limitations

    Performance is upper-bounded by the quality of the backbone outputs, requiring careful training-data selection for smaller-scale models.

Abstract

from arXiv · show

Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX.

1 Introduction

Perspective-trained foundation models produce distorted reconstructions and segmentations on fisheye images because radial distortion creates a feature-space covariate shift. DEX addresses this shift by aligning fisheye embeddings with perspective embeddings while preserving pretrained backbones.

  • Motivation: Perspective-camera training biases foundation models toward projective geometry assumptions.These models are trained predominantly on large-scale datasets captured with standard perspective cameras.
  • Motivation: Increasing field of view produces stronger radial warping near image borders and disrupts local pixel arrangements.The resulting mismatch changes encoded features and degrades downstream estimates.
  • Motivation: Existing rectification methods preserve perspective-model compatibility but require calibration or resampling and can introduce artifacts.Reported artifacts include stretching, seams, aliasing, and gaps between patches.
  • Proposed direction: DEX modulates fisheye latent embeddings toward the distribution that produces high-fidelity perspective estimates.The approach avoids explicitly modeling camera geometry in the input and instead adjusts embedding values.
  • Proposed direction: DEX is designed to be architecture- and task-agnostic across convolutional and Transformer models, depth estimation, and open-vocabulary segmentation.The contribution also reports zero-shot evaluation on ScanNet++, KITTI-360, and WoodScape.

2 Related Works

Prior work addresses wide-FoV vision through projection changes, camera-specific training, fine-tuning, and large pretrained models, but these strategies trade flexibility, data efficiency, or generality. DEX differs by supporting architectures and tasks beyond Transformer-specific or depth-specific adaptation.

  • Transformation-based solutions: Projection-based methods transform unrectified images into equirectangular, tangent, spherical, or cube-map representations.These approaches can suffer from resampling artifacts and projection-specific distortions.
  • Camera-specific methods: Camera-specific depth estimators avoid canonical projections but face limited wide-FoV training data and specialized designs.The scarcity of wide-FoV datasets makes training foundation models for these cameras difficult.
  • Camera-specific methods: Fine-tuning pretrained models can improve target-camera performance but risks parameter drift and loss of prior generalizability.The cited concern includes generalization across 3D scenes and previously learned cameras.
  • Generalizable estimators: Generalizable depth estimators combine mixed-camera training with projection mechanisms in inputs or latent representations.The related methods extend foundation-model depth estimation across cameras using canonical spaces.
  • Differentiation from prior adaptation: Compared with Calibration Tokens, DEX supports CNNs and Transformers, uses a task-agnostic loss, and selects convex extender combinations without token removal.Calibration Tokens are limited to Transformers and use a depth-specific loss.
  • Differentiation from prior adaptation: Unlike cross-attention and LoRA, DEX uses feature-dependent weights to select convex combinations of learned Extender modulators.The selection mechanism directly determines the modulation applied to encoder features.

3 Method

DEX inserts lightweight modulators into a frozen perspective-trained encoder and learns them by aligning synthetically distorted fisheye outputs with perspective references. Convex combinations of Extenders adapt latent features, while task-specific objectives support depth and segmentation.

  • DEX architecture: DEX models fisheye-induced latent covariate shift by translating fisheye embeddings toward perspective embeddings.Its Extenders represent distortion-related modulation in feature space rather than rectifying image pixels.
  • DEX architecture: Each Extender acts as a discrete center, and convex combinations provide continuous distortion corrections selected from the input features.The architecture includes a soft selection mechanism and multiple Extender modulation parameters.
  • DEX architecture: DEX modulates encoder outputs at every layer and feeds the resulting embeddings into subsequent layers and skip connections.The same mechanism applies to token or feature-vector representations.
  • Training: Training freezes the perspective backbone, synthetically distorts calibrated perspective images, inverse-aligns fisheye outputs, and minimizes an alignment loss.The procedure is self-supervised and uses perspective outputs or features as references.
  • Task objectives: For monocular depth estimation, DEX converts Cartesian depth supervision into spherical range and optimizes the aligned range prediction.This parameterization is intended for high-field-of-view settings.
  • Task objectives: For open-vocabulary segmentation, DEX compares perspective and fisheye image embeddings before their similarity with language embeddings assigns pixel classes.The objective is formulated in latent space rather than over the full network output.
  • Inference: At inference, DEX handles fisheye images without rectification or calibration knowledge and produces depth or image embeddings for the baseline decoder.Segmentation embeddings are subsequently compared with language embeddings using the baseline architecture.

4 Experiments

DEX is evaluated zero-shot on real indoor and outdoor fisheye datasets across monocular depth estimation and open-vocabulary segmentation. It improves baseline performance and extends to multitask depth and segmentation.

  • Evaluation Setup: Training uses synthesized KB-distorted fisheye images from calibrated perspective data, without requiring ground-truth or real fisheye images.The same Extender set is trained for both indoor and outdoor evaluation domains.
  • Evaluation Setup: DEX is tested with Transformer- and CNN-based depth models, alongside LSeg and SED for open-vocabulary segmentation.The evaluation includes MiDaS, DepthAnything, UniDepthV2, VNL, LSeg, and SED.
  • Indoor MDE: 19% lower RMSE and 15% higher δ1 are achieved on average over LoRA and Calibration Tokens for indoor monocular depth estimation.On ScanNet++, DEX also improves over UniK3D by 8% averaged over the metrics when applied to UniDepthV2.
  • Outdoor MDE: 11% improvement in RMSE and 11% improvement in δ1 are obtained on average over LoRA and Calibration Tokens for outdoor monocular depth estimation.KITTI-360 has higher FoV and more severe distortion; spherical prediction is especially helpful there.
  • Open-Vocabulary Segmentation: Around 13% higher mIoU than Calibration Tokens is reported for open-vocabulary segmentation on WoodScape.The evaluation uses LSeg and SED with queried classes including road, lane markings, curb, humans, vehicles, and street signs.
  • DEX as Task-Agnostic: DEX improves both depth estimation on KITTI-360 and segmentation on WoodScape when applied to the multitask PanopticDepth framework.The experiment jointly optimizes depth and segmentation losses using the same training set and baseline outputs as supervision.

5 Analysis

The analysis examines DEX’s feature alignment, distortion decoding, ablations, and computational overhead. Results indicate that DEX aligns fisheye features with perspective features while modeling distortion-related information with minimal added complexity.

  • Ablation Study: Removing spherical-coordinate extension or low-rank decomposition degrades depth-estimation performance, with Cartesian-coordinate learning especially harmful outdoors.The outdoor setting has wider FoV and stronger fisheye distortion; removing low-rank decomposition also introduces over-parameterization.
  • Feature Alignment: DEX aligns fisheye image features closely with corresponding perspective features in t-SNE visualizations.This supports feature-space alignment as a mechanism for adapting perspective-trained models to fisheye inputs.
  • Computational Overhead: DEX adds minimal computational complexity over the backbone models.The overhead analysis considers additional memory and inference time.
  • DEX Decoding: k_1 is decoded most accurately, while later coefficients such as k_4 have lower accuracy and wider uncertainty.The degradation follows the visual influence of the coefficients: earlier coefficients produce larger distortion changes than later ones.
  • DEX Decoding: Perturbing distortion coefficients by the method’s prediction error causes very little visual difference in generated fisheye images.This indicates that recovering distortion coefficients with higher fidelity is difficult, especially for changes with limited visual effect.

6 Discussion

DEX adapts perspective-trained vision foundation models to fisheye imagery in feature space without backbone modification or explicit test-time rectification. The framework is reusable across architectures and tasks, while its performance remains bounded by the backbone’s output quality.

  • Discussion: DEX adapts perspective-trained models to fisheye images without modifying the backbone or explicitly rectifying geometry at test time.It operates directly in feature space to mitigate projection mismatch from high-FoV lenses.
  • Discussion: DEX activations can be decoded into KB distortion coefficients, supporting camera-calibration analysis.The decoding procedure predicts distortion parameters, although later coefficients are represented with wider uncertainty.
  • Discussion: The self-supervised alignment framework is reusable across architectures and tasks.It combines a frozen backbone, synthetic domain shift, inverse geometric alignment, and lightweight Extenders.
  • Limitations: DEX performance is upper-bounded by the quality of the backbone models used as supervision.Training-data selection therefore matters, particularly for smaller-scale backbone models whose outputs may be less reliable.

SUPPLEMENTARY MATERIAL

The supplementary analysis visualizes how DEX distributes its modulation across fisheye image features. Extenders show distinct regional affinities, indicating coordinated feature-space adaptation.

  • Affinity Maps: Each Distortion Extender exhibits different affinity toward regions of fisheye image features.This selective modulation is also described as cooperative alignment toward perspective-image features.

B Sensitivity Studies

Sensitivity studies examine DEX hyperparameters, layer placement, and label wording. DEX improves segmentation across label sets, while label-dependent variation is attributed primarily to the LSeg base model.

  • Hyperparameters and Layer Placement: The sensitivity study varies the number of Extenders, decomposition dimension, and layer-placement strategy for UniDepthV2.Compared placements include every other layer, the first layer, and a shared set reused across layers.
  • Label Wording: DEX consistently improves open-vocabulary segmentation performance regardless of the evaluated label set.The label-wording study evaluates LSeg with DEX on WoodScape.
  • Label Wording: Performance differences across label sets are attributed to the LSeg base model rather than inherent sensitivity in DEX.The study treats label-set variation as an artifact of the base model’s behavior.

C Implementation Details

The implementation converts fisheye distance estimates between spherical range and Cartesian depth using camera intrinsics and distortion-model inversions.

  • Depth experiments use 12 Distortion Extenders per transformer block, while VNL uses 72, 18, 9, 4, and 2 across five encoder residual blocks.
  • For segmentation, DEX uses 24 Extenders per Transformer block and 24, 12, 6, and 3 across CNN encoder resolutions.
  • All experiments learn projection matrices with inner dimension d = 20.
  • D.1 R to z Transformation: To recover Cartesian depth from an R fisheye map, the method inverts intrinsics and the fisheye transformation to obtain the undistorted projection direction.
  • D.1 R to z Transformation: The recovered projection direction is scaled by spherical range R, and its z component becomes the pixel depth.
  • D.1 R to z Transformation: For the reverse transformation, the method recovers the projection direction, scales by z, and takes its magnitude to obtain R.

E Dataset Details

The experiments combine diverse perspective training data with real indoor and outdoor fisheye evaluation datasets, while documenting sampling, metrics, and qualitative reconstructions.

  • Training Datasets: NYUv2 provides roughly 400k indoor RGB–depth pairs from 464 scenes at 640×480 resolution.
  • Training Datasets: IRS, VOID, Hypersim, and Waymo broaden training coverage across synthetic interiors, real indoor spaces, photorealistic scenes, and outdoor driving.IRS contributes up to 103,316 frames, VOID around 58k pairs, Hypersim approximately 77k pairs, and Waymo about 230k frames.
  • Testing Datasets: Evaluation focuses on real fisheye or wide-FOV datasets, including indoor ScanNet++ and outdoor KITTI-360.ScanNet++ supplies fisheye depth ground truth, while KITTI-360 includes panoramic fisheye cameras and high-resolution LiDAR.
  • Testing Datasets: WoodScape provides four surround-view fisheye cameras with full 360° coverage and multimodal automotive measurements.
  • Training Protocols: The 200k depth-training mix samples 25k NYUv2, 60k IRS, 30k VOID, 60k Hypersim, and 25k Waymo examples.
  • Training Protocols: The 80k depth mix excludes IRS and Hypersim because synthetic data introduced a sim-to-real gap that degraded training accuracy.
  • Qualitative Results: Additional DEX reconstructions recover curved and straight roads outdoors and farther room regions and straight edges indoors.

G Evaluation Metrics

Evaluation uses standard depth and segmentation metrics, and compares DEX with undistortion preprocessing using point-cloud accuracy and completeness measures.

  • Depth Estimation: RMSE measures linear-space depth error, where lower values indicate better accuracy, while δ1 reports threshold-based accuracy.
  • Undistortion Comparison: DEX substantially improves over undistortion preprocessing on ScanNet++ and KITTI-360 using Chamfer distance and F1 score.The comparison changes metrics because undistortion can alter effective field of view and reduce scene coverage.

I Extended Discussion on Limitations and Future Work

DEX remains bounded by its baseline model: domain-specific baseline checkpoints provide weaker supervision when applied outside their training domain.

  • Limitations: DEX performance is bounded by the performance of the baseline model.
  • Limitations: Indoors-trained VNL degrades on outdoor data, and outdoors-trained VNL degrades on indoor data, because each checkpoint is domain-specific.Out-of-distribution use weakens the supervision signal and yields less improvement in the mismatched domain.
  • Discussion: DEX is motivated by extending pretrained models through latent feature adjustment rather than explicitly modeling camera geometry in input and latent coordinates.
Loading 2608.27860v1…