PaperlayerView exact revision on arXiv

Source-linked AI summary

UniSHARP: Universal Sharp Monocular View Synthesis

Meixi Song, Dizhe Zhang, Hao Ren, Ruiyang Zhang, Bo Du, Ming-Hsuan Yang, Lu Qi

arXiv:2606.07514v1cs.CV

TL;DR

Monocular Gaussian regressors trained on narrow-FoV perspective images do not reliably generalize to wide-FoV, fisheye, and panoramic cameras. UniSHARP aligns feature and Gaussian prediction in a shared omnidirectional representation, achieving state-of-the-art rendering quality and strong cross-camera generalization on a FoV-stratified benchmark.

  • Problem

    Perspective-trained monocular Gaussian regressors do not reliably generalize to wide-FoV, fisheye, and panoramic cameras, limiting their applicability across heterogeneous imaging systems.

    Source: §1 · p. 2

  • Method

    UniSHARP aligns feature and Gaussian prediction in a shared omnidirectional representation, organizing Gaussian primitives in ray-distance space and decoding 2D semantic and 3D spatial features.

    Source: §1 · p. 2

  • Results

    UniSHARP achieves state-of-the-art rendering quality and strong cross-camera generalization on a benchmark spanning perspective, wide-FoV, fisheye, and panoramic cameras.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The FoV-stratified benchmark enables systematic assessment of rendering quality as camera field of view increases from 60° to 360°.

    Source: §1 · p. 2

    When target views expose large regions absent from the source image, UniSHARP has limited evidence to hallucinate unseen content, so holes or weakly supported structures may appear near outer boundaries.

    Source: B Limitations · p. 16

Abstract

from arXiv · show

In this work, we focus on extending SHARP, the popular photorealistic view synthesis method, for universal monocular rendering across a continuum of camera systems, from conventional perspective cameras to wide-field-of-view, fisheye and omnidirectional panoramic settings. To overcome the pinhole-specific assumptions of SHARP, our key idea is to align various images in a unified omnidirectional latent space. Thus, we propose UniSHARP, which performs implicit alignment in both feature and Gaussian spaces. Specifically, Gaussian primitives are arranged along rays and radial distances in a ray-based universal representation, while 2D semantic and 3D spatial features extracted from UniK3D-inspired encoders are jointly decoded to generate the complete Gaussian cloud. To comprehensively evaluate our method, we construct a benchmark covering diverse imaging systems across various scenes. The benchmark is further stratified by field of view (FoV) to enable fine-grained assessment of the universal monocular rendering task. Extensive experiments on the proposed benchmark demonstrate the effectiveness of UniSHARP, outperforming alternative methods by a large margin. The project page can be found at: https://insta360-research-team.github.io/Unisharp-website/

1 Introduction

Single-image view synthesis remains ill-posed, and monocular methods trained on narrow-FoV perspective images do not generalize to diverse camera systems. UniSHARP extends SHARP with a unified omnidirectional representation for universal monocular rendering.

Introduction

  • UniSHARP extends SHARP across perspective, wide-FoV, fisheye, and panoramic inputs by predicting Gaussians in shared ray-distance space and aligning features implicitly.The representation organizes Gaussian primitives along rays and radial distances, rather than projection-specific image coordinates.Source: §1 · p. 2
  • When camera parameters are unavailable, UniSHARP uses predicted ray fields to infer camera type and rendering geometry from a single RGB image.This allows the same feedforward Gaussian predictor to operate without manually supplied intrinsics.Source: §1 · p. 2
  • Its feature-space prediction pipeline fuses 2D semantic encodings with 3D spatial features and allocates Gaussians at native input resolution.This preserves geometric priors and high-frequency image details without camera-specific resizing.Source: §1 · p. 2
  • For panoramic inputs, UniSHARP uses spherical Gaussian initialization and distortion-aware probabilistic dropout to regularize Gaussians under severe equirectangular distortion.Source: §1 · p. 2
  • The FoV-stratified benchmark spans real and simulated perspective, wide-FoV, fisheye, and panoramic data from 60° to 360°; UniSHARP demonstrates state-of-the-art rendering quality and strong cross-camera generalization.It combines established validation datasets with OmniRooms, an AirSim-based indoor panoramic dataset, and its projected wide-FoV variant.Source: §1 · p. 2

2 Related work

Prior NVS methods either rely on multiple views or address monocular inputs mainly under perspective-camera assumptions. UniSHARP targets feedforward monocular 3DGS across perspective, wide-FoV, fisheye, and panoramic cameras.

Multi-image novel view synthesis

  • Multi-image NVS spans NeRF [10] [26] [27], 3DGS [11], feedforward and sparse-view models [28, 29, 2, 3, 30], and large or pose-free systems [31] [33] [34], but depends on multi-view cues.UniSHARP instead predicts a renderable Gaussian representation from one image.Source: §2 · p. 3

Single-image novel view synthesis

  • Single-image NVS progressed from NeRF and MPI [35] [36] to layered methods [37] [38] [39] [40], reconstruction models [41] [42], monocular 3DGS [12] [13] [43], and diffusion [45] [46], yet mainly assumes perspective imagery.UniSHARP extends monocular 3DGS to perspective, wide-FoV, fisheye, and panoramic inputs.Source: §2 · p. 3

Wide-FoV novel view synthesis

  • Wide-FoV methods use ray-based geometry [47], adapted 3DGS [14] [48], self-calibration [50], or panoramic representations [16] [17], but often require optimized scenes, calibration, or multiple observations.UniSHARP unifies these camera settings in one feedforward 3DGS model.Source: §2 · pp. 3–4

3 Method

UniSHARP decouples camera projection from scene representation by predicting 3D Gaussians in a unified ray-distance space. Its pipeline combines geometry anchors, feature-conditioned residuals, mixed-camera training, and camera-geometry recovery for pose-free inference.

3.1 Ray Based Universal Representation

  • A predicted unit ray and radial distance define each 3D point as x_p = d_p r_p, giving Gaussian attributes a consistent coordinate system across camera models.Unlike image-plane coordinates, the ray-distance representation separates viewing direction from scene range and measures spatial footprints along rays, supporting initialization and refinement across projections.Source: §3.1 · p. 4

3.2 Model Design

  • The model composes a camera-unified initialization of Geometry Anchored Gaussians with Feature Conditioned Gaussian residuals to produce the final renderable Gaussians.The residuals are predicted from 2D semantic and 3D ray-based features.Source: §3.2 · p. 4

3.2.1 Geometry Anchored Gaussians

  • Two Geometry Anchored Gaussian layers use native-ray sampling: the first aligns with the visible surface, while the second represents disocclusions and high-frequency structures.An additional depth head predicts the second layer’s separate radial-distance hypothesis, which can specialize through rendering supervision and regularization.Source: §3.2.1 · p. 4

3.2.2 Feature Conditioned Gaussian Residuals

  • Feature Conditioned Gaussian residuals fuse 2D semantic image features with 3D ray-based geometric features and adjust anchor centers, inverse distance, scale, quaternion, color, and opacity.A Gaussian decoder predicts the residual tensor for each geometry-anchored location, and the final Gaussian is composed from the anchor and residual.Source: §3.2.2 · p. 5

3.3 Training Strategy and Objective

  • UniSHARP trains one shared architecture across perspective, wide-FoV, fisheye, and panoramic data, using a common ray-based interface without camera-specific branches.For panoramas, latitude-dependent dropout selectively suppresses the second Gaussian layer while preserving the first for visible-surface coverage; appearance, depth, and Gaussian-regularization terms supervise training.Source: §3.3 · pp. 5–6

3.4 Extension to Pose-Free Model

  • For pose-free inference, UniSHARP estimates camera geometry from predicted rays and uses the recovered camera for Gaussian initialization and novel-view rendering, retaining the shared predictor.It fits pinhole intrinsics or fisheye parameters for perspective and fisheye inputs and uses a deterministic spherical camera for panoramas, enabling orbit and forward-motion views from an uncalibrated image.Source: §3.4 · p. 6

4 Experiments

The experiments introduce an FoV-stratified benchmark for universal monocular novel-view synthesis and report strong performance across perspective, wide-FoV, fisheye, and panoramic cameras.

4.2 Qualitative and Quantitative Evaluation

  • For perspective synthesis, UniSHARP produces sharper target-view geometry, fewer disocclusion artifacts, and view-consistent scene structure than perspective monocular baselines.The qualitative comparison is shown in Figure 3.Source: Figure 3 · p. 7

4.1 Benchmark & Metrics.

  • The FoV-stratified benchmark diagnoses rendering behavior from narrow perspective through full 360° equirectangular inputs using a unified protocol for four camera families.Evaluation uses the first sequence frame as source and the next ten as targets, retaining views with over 60% overlap, under 0.5 m camera-center distance, and under 10-frame index gap; PSNR, SSIM, and LPIPS are averaged over valid targets.Source: §4.1 · pp. 7–8

4.2 Qualitative and Quantitative Evaluation

  • UniSHARP improves all three metrics on OmniRooms-Wide, outperforms baselines on ScanNet++ Fisheye, and achieves the best PSNR, SSIM, and LPIPS on HM3D, OmniRooms, and Replica.The reported results cover projected wide-FoV, native fisheye, and panoramic inputs, including out-of-domain Replica.Source: Table 3 · p. 8, Table 4 · p. 9, §4.2 · p. 9
  • On zero-shot Tanks and Temples, UniSHARP achieves the best PSNR and LPIPS among the strongest baselines, with SSIM close to Flash3D.This out-of-domain evaluation indicates that the unified representation generalizes across datasets while improving overall reconstruction fidelity.Source: Table 5 · p. 9, §4.2 · p. 9
  • The pose-free WildRGB-D variant maintains competitive rendering quality by estimating camera geometry from predicted rays, without requiring camera calibration.The calibrated variant uses available camera parameters; the pose-free variant estimates the camera model and rendering geometry from the predicted ray field.Source: Table 6 · p. 9, §4.2 · p. 9

4.3 Ablation Studies

  • Ablations show that replacing learned semantic and geometric features with direct depth-RGB inputs causes the largest degradation, while removing the second Gaussian layer also hurts both datasets.Native-resolution allocation helps preserve input detail, and panoramic distortion adaptation has a clearer effect on HM3D than on perspective images.Source: Table 7 · p. 10, §4.3 · p. 10

5 Conclusion

UniSHARP extends monocular novel-view synthesis across heterogeneous camera systems by predicting Gaussians in a shared ray-distance space. On a FoV-stratified benchmark, it maintains strong perspective-dataset performance and substantially improves synthesis across diverse camera systems.

5 Conclusion

  • UniSHARP maintains strong performance on perspective datasets and substantially improves novel view synthesis across diverse camera systems.The benchmark covers real and simulated scenes from narrow perspective to full panoramic cameras.Source: §5 · p. 10
  • UniSHARP is a universal-camera feedforward 3DGS framework that predicts Gaussians in shared ray-distance space using Geometry Anchored Gaussians and Feature Conditioned Gaussian residuals.This design preserves single-image Gaussian regression efficiency while supporting perspective, wide-FoV, fisheye, and panoramic inputs in one model.Source: §5 · p. 10
  • The authors introduce a FoV-stratified benchmark for systematic evaluation across real and simulated scenes spanning narrow perspective to full panoramic cameras.They present the work as a practical foundation for monocular 3D Gaussian rendering beyond the pinhole camera model.Source: §5 · p. 10

A Additional Experiments and Ablations · A.1 Implementation Details

UniSHARP’s experiments use a feature-only architecture with a pretrained UniK3D ViT-L backbone, generic-camera rasterization for wide-FoV and fisheye rendering, and a specified Adam optimization schedule and loss weighting.

A.1 Implementation Details

  • Experiments were conducted on 8 H20 GPUs.Source: §A.1 · p. 14
  • UniSHARP uses the feature-only architecture described in Sec. 3.Source: §A.1 · p. 14
  • The UniK3D ViT-L backbone was initialized from pretrained weights [47].Source: §A.1 · p. 14
  • Wide-FoV and fisheye rendering use the 3DGEER generic-camera Gaussian rasterizer [62].Source: §A.1 · p. 14
  • The model was optimized with Adam [63] for 106 iterations, using a 104-iteration warmup followed by cosine learning-rate decay.Source: §A.1 · p. 14
  • Learning rates decay from 1.0 × 10−5 to 1.0 × 10−6 for the depth head, and from 1.2 × 10−4 to 1.6 × 10−5 for the Gaussian decoder and prediction modules.Source: §A.1 · p. 14
  • Loss weights are λc = 1.0, λa = 1.0, λp = 1.0, λd = 0.5, λtv = 1.0, λg = 0.5, and λgi = 0.5.Source: §A.1 · p. 14

A.2 Training Objective Ablation

The full training objective performs best overall, and ablations show that target rendered-depth supervision is especially important for cross-view geometry. Appearance loss, TV regularization, and floater suppression also contribute to rendering quality and stability.

A.2 Training Objective Ablation

  • The full objective performs best overall, scoring 21.56 PSNR and 0.143 LPIPS on WildRGB-D and 29.24 PSNR and 0.065 LPIPS on HM3D.Table 8 compares variants that each remove one loss term from the full objective.Source: Table 8 · p. 14, §A.2 · p. 14
  • Removing target rendered-depth supervision causes the largest PSNR drop, to 20.42 on WildRGB-D and 27.12 on HM3D, while LPIPS rises to 0.206 and 0.136, respectively.Source-side depth supervision alone cannot constrain the Gaussian scene after view transformation; target-depth supervision maintains cross-view geometry and suppresses view-dependent distortions.Source: §A.2 · p. 14
  • Perceptual appearance loss improves visual fidelity, while second-layer TV regularization and floater suppression stabilize the Gaussian field.Floater suppression is particularly important for panoramic scenes: removing it increases HM3D LPIPS from 0.065 to 0.153 due to unstable second-layer Gaussians near depth discontinuities.Source: §A.2 · pp. 14–15

A.3 Fisheye Dataset Visualization · A.4 Panoramic Inference via Cubemap Decomposition · A.5 Inference Time Comparison

The appendices visualize the wide angular coverage and radial distortion of fisheye data, show cubemap-based SHARP seams against UniSHARP’s coherent panoramic rendering, and compare inference times. UniSHARP’s single-pass design avoids per-scene optimization and iterative generation at inference.

A.3 Fisheye Dataset Visualization

  • Figure 5’s fisheye validation samples show strong radial distortion and wide angular coverage, distinguishing native fisheye novel view synthesis from standard perspective evaluation.Source: Figure 5 · p. 15

A.4 Panoramic Inference via Cubemap Decomposition

  • SHARP cannot directly ingest panoramas under its pinhole assumption; a cubemap workaround processes six faces independently, leaving Gaussian predictions inconsistent across face boundaries.Source: §A.4 · p. 15
  • Stitching cubemap-based SHARP renderings produces prominent seams and view-dependent discontinuities, especially near geometric edges and depth boundaries.Source: §A.4 · p. 15, Figure 6 · p. 16
  • On a representative panoramic sample, UniSHARP renders a coherent target view without seams, operating directly on panoramic input in unified ray-distance space rather than decomposing it into cubemap faces.Source: §A.4 · p. 15, Figure 6 · p. 16

A.5 Inference Time Comparison

  • UniSHARP takes 3.1 seconds to infer, compared with PanoDreamer at 8.6 seconds and Matrix3D at 38.8 seconds; they are 2.8× and 12.5× slower, respectively.Runtime is measured under the same evaluation setting.Source: Table 9 · p. 16, §A.5 · p. 16
  • UniSHARP predicts the complete Gaussian representation in a single feedforward pass and renders it directly, avoiding per-scene optimization, diffusion sampling, and iterative video generation at inference.Source: §A.5 · p. 16

B Limitations … D.6 Metric Implementation

UniSHARP supports universal-camera view synthesis, but its feedforward predictions have limited evidence for large unseen regions. The accompanying benchmark specifies dataset construction, camera metadata, evaluation, and metrics to support reproducible comparison.

B Limitations

  • Large unseen regions remain a limitation: UniSHARP handles moderate disocclusions but may produce holes or weakly supported structures near image boundaries.The model predicts Gaussians feedforward rather than completing scenes generatively; improving long-range extrapolation while retaining efficiency and geometric consistency remains future work.Source: B Limitations · p. 16

C Societal Impact

  • Universal-camera rendering may broaden single-image 3D applications, while the benchmark can support systematic evaluation of camera-general methods.Potential applications include embodied AI, robotics, AR/VR, telepresence, and spatial documentation; single-image inference may reduce capture requirements.Source: C Societal Impact · pp. 16–17

D Benchmark Details; D.1 Dataset Splits and Scene Selection

  • The benchmark uses fixed held-out evaluations, preserving scene-level separation and covering perspective, wide-FoV, fisheye, and panoramic datasets.RealEstate10K follows its official test split; Tanks and Temples and Replica serve as out-of-domain perspective and panoramic sets, respectively. OmniRooms-Wide uses 130° equidistant views, while ScanNet++ Fisheye retains native calibration.Source: D Benchmark Details · p. 17, §D.1 · p. 17

D.2 OmniRooms Construction

  • OmniRooms samples nearby panoramic views around indoor anchors to evaluate translation-induced synthesis with meaningful parallax and disocclusions.Anchors are sampled on a 0.5 m grid at heights from 60 to 180 cm; each produces 30 camera positions within a 30 cm cube. RGB, depth, and camera metadata are rendered, but depth is not a test-time input.Source: §D.2 · pp. 17–18

D.3 Source-Target Pair Filtering

  • Pairs are retained only when overlap is at least 60%, camera distance is below 0.5 m, and frame-index gap is at most 10.Overlap counts target pixels whose visible 3D points are also visible in the source, using depth, poses, or meshes and accounting for panoramic wrap-around and fisheye valid masks.Source: §D.3 · p. 18

D.4 Camera Metadata and Projection Models

  • Camera metadata preserves each dataset’s projection model while standardizing extrinsics as camera-to-world transforms.The benchmark includes pinhole perspective, equidistant wide-FoV and fisheye, and equirectangular panoramic projections; simulated fisheye views use 130° FoV at 1024 × 1024 resolution.Source: §D.4 · pp. 18–19

D.5 Evaluation Protocol

  • Evaluation uses one source image and target camera parameters when calibrated, then compares rendered target views with ground truth using PSNR, SSIM, and LPIPS.Scores are averaged over target views and then sequences; valid-pixel masks and circular horizontal handling are used where applicable.Source: §D.5 · p. 19

D.6 Metric Implementation

  • Metric implementation uses PSNR, RGB-channel SSIM, and AlexNet LPIPS, with camera-group averages computed as unweighted dataset-level means.SSIM uses an 11-pixel Gaussian window with standard deviation 1.5 and constants C1 = 0.012 and C2 = 0.032; LPIPS uses the official AlexNet implementation.Source: §D.6 · p. 20

D.7 Baseline Evaluation Details … D.10 Licenses, Ethics, and Privacy

The appendices specify how baseline, runtime, and pose-free evaluations are conducted, how benchmark samples are filtered, and how dataset licensing and privacy are handled.

D.7 Baseline Evaluation Details

  • All baselines use the same source-target pairs, official checkpoints and inference code when available, and the unchanged target-view protocol for compatible camera representations.When a method supports only some camera models, evaluation uses the closest compatible input representation.Source: §D.7 · p. 20
  • Perspective baselines use original pinhole parameters, while PanoDreamer and Matrix3D use their expected input representations and benchmark trajectories, with native-projection outputs reprojected when needed.Inputs and outputs follow each method’s official resizing; official default inference and post-processing settings are used unless direct target projection is unavailable.Source: §D.7 · p. 20
  • Runtime comparisons fix hardware, rendered-view count, batch size, resolution, and timing iterations, exclude data loading and decoding, and measure forward rendering after warm-up.Timing includes model forward computation and target-view rendering.Source: §D.7 · p. 20

D.8 Pose-Free Evaluation Details

  • In pose-free evaluation, UniSHARP receives only source RGB, predicts a ray field, infers the camera model, and recovers geometry; target camera parameters still define evaluation views.On WildRGB-D, source calibration is replaced with recovered geometry, while target-view definitions remain unchanged for comparison with ground truth.Source: §D.8 · pp. 20–21

D.9 Quality Control

  • Quality control excludes samples with missing images, invalid camera metadata, or required invalid depth; fisheye metrics use valid masks and no-extrapolation masks omit rendered black borders.Simulated OmniRooms samples with invalid depth buffers or severe clipping are removed, and released camera centers remain within 60–180 cm in Z.Source: §D.9 · p. 21

D.10 Licenses, Ethics, and Privacy

  • Existing datasets retain their original license terms, OmniRooms and OmniRooms-Wide use CC BY-NC 4.0 for research and non-commercial use, and released metadata excludes private user information.Source: §D.10 · p. 21

Related papers

Loading 2606.07514v1…