PaperlayerView exact revision on arXiv

Source-linked AI summary

From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Rit Gangopadhyay, Alex Wong

arXiv:2608.27860v1cs.CVcs.AI

TL;DR

Perspective-trained vision foundation models fail on fisheye imagery because radial distortion shifts latent features away from those learned from perspective images. DEX learns lightweight feature-space corrections with self-supervised alignment, improving depth estimation and open-vocabulary segmentation across fisheye settings while preserving frozen backbones. Its calibration decoding is useful but distortion-coefficient recovery remains difficult, and performance is bounded by backbone quality.

  • Problem

    Perspective-trained foundation models produce erroneous fisheye outputs because radial distortion induces a systematic shift in local pixels and latent embeddings.

    Source: §1 · pp. 2–3

  • Method

    DEX inserts lightweight learnable Extenders into a frozen backbone and aligns synthetically distorted fisheye embeddings with perspective embeddings using convex combinations and self-supervised loss.

    Source: §3 · pp. 6–7

  • Results

    DEX consistently improves monocular depth estimation and open-vocabulary segmentation across indoor and outdoor fisheye datasets and multiple backbone architectures.

    Source: §1 · p. 4, §6 · p. 15

  • Takeaways & Limitations

    Feature-space adaptation can extend perspective-trained models to fisheye imagery without explicit test-time rectification or backbone modification.

    Source: §6 · pp. 14–15

    Performance is upper-bounded by the quality of the backbone outputs, requiring careful training-data selection for smaller-scale models.

    Source: §6 · p. 15

Abstract

from arXiv · show

Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX.

1 Introduction

Perspective-trained foundation models produce distorted reconstructions and segmentations on fisheye images because radial distortion creates a feature-space covariate shift. DEX addresses this shift by aligning fisheye embeddings with perspective embeddings while preserving pretrained backbones.

Motivation

  • Perspective-camera training biases foundation models toward projective geometry assumptions.These models are trained predominantly on large-scale datasets captured with standard perspective cameras.Source: §1 · p. 1
  • Increasing field of view produces stronger radial warping near image borders and disrupts local pixel arrangements.The resulting mismatch changes encoded features and degrades downstream estimates.Source: §1 · p. 2
  • Existing rectification methods preserve perspective-model compatibility but require calibration or resampling and can introduce artifacts.Reported artifacts include stretching, seams, aliasing, and gaps between patches.Source: §1 · p. 2

Proposed direction

  • DEX modulates fisheye latent embeddings toward the distribution that produces high-fidelity perspective estimates.The approach avoids explicitly modeling camera geometry in the input and instead adjusts embedding values.Source: §1 · p. 3
  • DEX is designed to be architecture- and task-agnostic across convolutional and Transformer models, depth estimation, and open-vocabulary segmentation.The contribution also reports zero-shot evaluation on ScanNet++, KITTI-360, and WoodScape.Source: §1 · pp. 3–4

2 Related Works

Prior work addresses wide-FoV vision through projection changes, camera-specific training, fine-tuning, and large pretrained models, but these strategies trade flexibility, data efficiency, or generality. DEX differs by supporting architectures and tasks beyond Transformer-specific or depth-specific adaptation.

Transformation-based solutions

  • Projection-based methods transform unrectified images into equirectangular, tangent, spherical, or cube-map representations.These approaches can suffer from resampling artifacts and projection-specific distortions.Source: §2 · p. 4

Camera-specific methods

  • Camera-specific depth estimators avoid canonical projections but face limited wide-FoV training data and specialized designs.The scarcity of wide-FoV datasets makes training foundation models for these cameras difficult.Source: §2 · pp. 4–5
  • Fine-tuning pretrained models can improve target-camera performance but risks parameter drift and loss of prior generalizability.The cited concern includes generalization across 3D scenes and previously learned cameras.Source: §2 · p. 5

Generalizable estimators

  • Generalizable depth estimators combine mixed-camera training with projection mechanisms in inputs or latent representations.The related methods extend foundation-model depth estimation across cameras using canonical spaces.Source: §2 · p. 5

Differentiation from prior adaptation

  • Compared with Calibration Tokens, DEX supports CNNs and Transformers, uses a task-agnostic loss, and selects convex extender combinations without token removal.Calibration Tokens are limited to Transformers and use a depth-specific loss.Source: §2 · pp. 5–6
  • Unlike cross-attention and LoRA, DEX uses feature-dependent weights to select convex combinations of learned Extender modulators.The selection mechanism directly determines the modulation applied to encoder features.Source: §2 · p. 6

3 Method

DEX inserts lightweight modulators into a frozen perspective-trained encoder and learns them by aligning synthetically distorted fisheye outputs with perspective references. Convex combinations of Extenders adapt latent features, while task-specific objectives support depth and segmentation.

DEX architecture

  • DEX models fisheye-induced latent covariate shift by translating fisheye embeddings toward perspective embeddings.Its Extenders represent distortion-related modulation in feature space rather than rectifying image pixels.Source: §3 · p. 6, Figure 2 · p. 7
  • Each Extender acts as a discrete center, and convex combinations provide continuous distortion corrections selected from the input features.The architecture includes a soft selection mechanism and multiple Extender modulation parameters.Source: §3 · pp. 7–8
  • DEX modulates encoder outputs at every layer and feeds the resulting embeddings into subsequent layers and skip connections.The same mechanism applies to token or feature-vector representations.Source: §3 · pp. 7–8

Training

  • Training freezes the perspective backbone, synthetically distorts calibrated perspective images, inverse-aligns fisheye outputs, and minimizes an alignment loss.The procedure is self-supervised and uses perspective outputs or features as references.Source: §3 · p. 8

Task objectives

  • For monocular depth estimation, DEX converts Cartesian depth supervision into spherical range and optimizes the aligned range prediction.This parameterization is intended for high-field-of-view settings.Source: §3 · p. 8
  • For open-vocabulary segmentation, DEX compares perspective and fisheye image embeddings before their similarity with language embeddings assigns pixel classes.The objective is formulated in latent space rather than over the full network output.Source: §3 · pp. 8–9

Inference

  • At inference, DEX handles fisheye images without rectification or calibration knowledge and produces depth or image embeddings for the baseline decoder.Segmentation embeddings are subsequently compared with language embeddings using the baseline architecture.Source: §3 · p. 9

4 Experiments

DEX is evaluated zero-shot on real indoor and outdoor fisheye datasets across monocular depth estimation and open-vocabulary segmentation. It improves baseline performance and extends to multitask depth and segmentation.

Evaluation Setup

  • Training uses synthesized KB-distorted fisheye images from calibrated perspective data, without requiring ground-truth or real fisheye images.The same Extender set is trained for both indoor and outdoor evaluation domains.Source: §4 · p. 9
  • DEX is tested with Transformer- and CNN-based depth models, alongside LSeg and SED for open-vocabulary segmentation.The evaluation includes MiDaS, DepthAnything, UniDepthV2, VNL, LSeg, and SED.Source: §4 · p. 9

Indoor MDE

  • 19% lower RMSE and 15% higher δ1 are achieved on average over LoRA and Calibration Tokens for indoor monocular depth estimation.On ScanNet++, DEX also improves over UniK3D by 8% averaged over the metrics when applied to UniDepthV2.Source: §4.1 · p. 11

Outdoor MDE

  • 11% improvement in RMSE and 11% improvement in δ1 are obtained on average over LoRA and Calibration Tokens for outdoor monocular depth estimation.KITTI-360 has higher FoV and more severe distortion; spherical prediction is especially helpful there.Source: §4.1 · p. 11

Open-Vocabulary Segmentation

  • Around 13% higher mIoU than Calibration Tokens is reported for open-vocabulary segmentation on WoodScape.The evaluation uses LSeg and SED with queried classes including road, lane markings, curb, humans, vehicles, and street signs.Source: §4.1 · p. 11

DEX as Task-Agnostic

  • DEX improves both depth estimation on KITTI-360 and segmentation on WoodScape when applied to the multitask PanopticDepth framework.The experiment jointly optimizes depth and segmentation losses using the same training set and baseline outputs as supervision.Source: §4.2 · pp. 12–13

5 Analysis

The analysis examines DEX’s feature alignment, distortion decoding, ablations, and computational overhead. Results indicate that DEX aligns fisheye features with perspective features while modeling distortion-related information with minimal added complexity.

Ablation Study

  • Removing spherical-coordinate extension or low-rank decomposition degrades depth-estimation performance, with Cartesian-coordinate learning especially harmful outdoors.The outdoor setting has wider FoV and stronger fisheye distortion; removing low-rank decomposition also introduces over-parameterization.Source: §5 · p. 13, Table 4 · p. 14

Feature Alignment

  • DEX aligns fisheye image features closely with corresponding perspective features in t-SNE visualizations.This supports feature-space alignment as a mechanism for adapting perspective-trained models to fisheye inputs.Source: §5 · p. 13, Figure 5 · p. 14

Computational Overhead

  • DEX adds minimal computational complexity over the backbone models.The overhead analysis considers additional memory and inference time.Source: Table 5 · p. 14

DEX Decoding

  • k_1 is decoded most accurately, while later coefficients such as k_4 have lower accuracy and wider uncertainty.The degradation follows the visual influence of the coefficients: earlier coefficients produce larger distortion changes than later ones.Source: §5 · p. 14
  • Perturbing distortion coefficients by the method’s prediction error causes very little visual difference in generated fisheye images.This indicates that recovering distortion coefficients with higher fidelity is difficult, especially for changes with limited visual effect.Source: §5 · p. 14

6 Discussion

DEX adapts perspective-trained vision foundation models to fisheye imagery in feature space without backbone modification or explicit test-time rectification. The framework is reusable across architectures and tasks, while its performance remains bounded by the backbone’s output quality.

Discussion

  • DEX adapts perspective-trained models to fisheye images without modifying the backbone or explicitly rectifying geometry at test time.It operates directly in feature space to mitigate projection mismatch from high-FoV lenses.Source: §6 · p. 14, Figure 6 · p. 15
  • DEX activations can be decoded into KB distortion coefficients, supporting camera-calibration analysis.The decoding procedure predicts distortion parameters, although later coefficients are represented with wider uncertainty.Source: Table 6 · p. 15, Figure 6 · p. 15
  • The self-supervised alignment framework is reusable across architectures and tasks.It combines a frozen backbone, synthetic domain shift, inverse geometric alignment, and lightweight Extenders.Source: §6 · p. 15

Limitations

  • DEX performance is upper-bounded by the quality of the backbone models used as supervision.Training-data selection therefore matters, particularly for smaller-scale backbone models whose outputs may be less reliable.Source: §6 · p. 15

SUPPLEMENTARY MATERIAL

The supplementary analysis visualizes how DEX distributes its modulation across fisheye image features. Extenders show distinct regional affinities, indicating coordinated feature-space adaptation.

Affinity Maps

B Sensitivity Studies

Sensitivity studies examine DEX hyperparameters, layer placement, and label wording. DEX improves segmentation across label sets, while label-dependent variation is attributed primarily to the LSeg base model.

Hyperparameters and Layer Placement

  • The sensitivity study varies the number of Extenders, decomposition dimension, and layer-placement strategy for UniDepthV2.Compared placements include every other layer, the first layer, and a shared set reused across layers.Source: B Sensitivity Studies · p. 22, Table 7 · p. 23

Label Wording

  • DEX consistently improves open-vocabulary segmentation performance regardless of the evaluated label set.The label-wording study evaluates LSeg with DEX on WoodScape.Source: B Sensitivity Studies · p. 23, Table 8 · p. 23
  • Performance differences across label sets are attributed to the LSeg base model rather than inherent sensitivity in DEX.The study treats label-set variation as an artifact of the base model’s behavior.Source: B Sensitivity Studies · p. 23, Table 8 · p. 23

C Implementation Details

The implementation converts fisheye distance estimates between spherical range and Cartesian depth using camera intrinsics and distortion-model inversions.

D.1 R to z Transformation

  • To recover Cartesian depth from an R fisheye map, the method inverts intrinsics and the fisheye transformation to obtain the undistorted projection direction.Source: §D.1 · p. 24
  • The recovered projection direction is scaled by spherical range R, and its z component becomes the pixel depth.Source: §D.1 · p. 24
  • For the reverse transformation, the method recovers the projection direction, scales by z, and takes its magnitude to obtain R.Source: §D.2 · p. 25

E Dataset Details

The experiments combine diverse perspective training data with real indoor and outdoor fisheye evaluation datasets, while documenting sampling, metrics, and qualitative reconstructions.

Training Datasets

  • NYUv2 provides roughly 400k indoor RGB–depth pairs from 464 scenes at 640×480 resolution.Source: E Dataset Details · p. 25
  • IRS, VOID, Hypersim, and Waymo broaden training coverage across synthetic interiors, real indoor spaces, photorealistic scenes, and outdoor driving.IRS contributes up to 103,316 frames, VOID around 58k pairs, Hypersim approximately 77k pairs, and Waymo about 230k frames.Source: E Dataset Details · p. 25

Testing Datasets

  • Evaluation focuses on real fisheye or wide-FOV datasets, including indoor ScanNet++ and outdoor KITTI-360.ScanNet++ supplies fisheye depth ground truth, while KITTI-360 includes panoramic fisheye cameras and high-resolution LiDAR.Source: E Dataset Details · p. 25
  • WoodScape provides four surround-view fisheye cameras with full 360° coverage and multimodal automotive measurements.Source: E Dataset Details · p. 25

Training Protocols

  • The 200k depth-training mix samples 25k NYUv2, 60k IRS, 30k VOID, 60k Hypersim, and 25k Waymo examples.Source: E Dataset Details · p. 26
  • The 80k depth mix excludes IRS and Hypersim because synthetic data introduced a sim-to-real gap that degraded training accuracy.Source: E Dataset Details · p. 26

Qualitative Results

G Evaluation Metrics

Evaluation uses standard depth and segmentation metrics, and compares DEX with undistortion preprocessing using point-cloud accuracy and completeness measures.

Depth Estimation

  • RMSE measures linear-space depth error, where lower values indicate better accuracy, while δ1 reports threshold-based accuracy.Source: G Evaluation Metrics · p. 27

Undistortion Comparison

I Extended Discussion on Limitations and Future Work

DEX remains bounded by its baseline model: domain-specific baseline checkpoints provide weaker supervision when applied outside their training domain.

Limitations

Discussion

Related papers

Loading 2608.27860v1…