PaperlayerView exact revision on arXiv

Source-linked AI summary

UniSHARP: Universal Sharp Monocular View Synthesis

Meixi Song, Dizhe Zhang, Hao Ren, Ruiyang Zhang, Bo Du, Ming-Hsuan Yang, Lu Qi

arXiv:2606.07514v1cs.CV

TL;DR

Perspective-trained monocular Gaussian regressors struggle across heterogeneous camera systems. UniSHARP uses shared ray-distance and feature spaces for universal view synthesis, achieving state-of-the-art rendering quality and strong cross-camera generalization on a FoV-stratified benchmark.

  • Problem

    Perspective-trained monocular Gaussian regressors fail to generalize reliably from narrow-FoV images to wide-FoV, fisheye, and panoramic camera systems.

  • Method

    UniSHARP predicts Gaussian primitives in shared ray-distance and feature spaces, combining geometric and semantic-spatial priors across heterogeneous camera systems.

  • Results

    UniSHARP demonstrates state-of-the-art rendering quality and strong cross-camera generalization on a FoV-stratified benchmark spanning perspective, wide-FoV, fisheye, and panoramic cameras.

  • Takeaways & Limitations

    The FoV-stratified benchmark enables controlled assessment of rendering quality across camera fields of view from 60° to 360°.

  • Takeaways & Limitations

    UniSHARP has limited evidence for hallucinating large unseen regions, which can produce holes or weakly supported structures near outer boundaries.

Abstract

from arXiv · show

In this work, we focus on extending SHARP, the popular photorealistic view synthesis method, for universal monocular rendering across a continuum of camera systems, from conventional perspective cameras to wide-field-of-view, fisheye and omnidirectional panoramic settings. To overcome the pinhole-specific assumptions of SHARP, our key idea is to align various images in a unified omnidirectional latent space. Thus, we propose UniSHARP, which performs implicit alignment in both feature and Gaussian spaces. Specifically, Gaussian primitives are arranged along rays and radial distances in a ray-based universal representation, while 2D semantic and 3D spatial features extracted from UniK3D-inspired encoders are jointly decoded to generate the complete Gaussian cloud. To comprehensively evaluate our method, we construct a benchmark covering diverse imaging systems across various scenes. The benchmark is further stratified by field of view (FoV) to enable fine-grained assessment of the universal monocular rendering task. Extensive experiments on the proposed benchmark demonstrate the effectiveness of UniSHARP, outperforming alternative methods by a large margin. The project page can be found at: https://insta360-research-team.github.io/Unisharp-website/

1 Insta360 Research 2 Sun Yat-sen University 3 Beihang University

The work is affiliated with six institutions and presents UniSHARP for monocular novel view synthesis across diverse camera types.

  • Affiliations: The authors are affiliated with Insta360 Research, Sun Yat-sen University, Beihang University, Wuhan University, and the University of California, Merced.The supplied passages list five institutions; the section title additionally identifies three numbered affiliations among them.
  • Method overview: UniSHARP takes a single perspective, wide-FoV, fisheye, or panoramic image and predicts a 3D Gaussian point cloud for high-quality novel-view rendering.This figure caption describes the method’s input camera types, predicted representation, and rendering objective.
  • Author notation: An asterisk denotes the corresponding author.

1 Introduction

UniSHARP addresses the failure of perspective-trained monocular 3DGS regressors on diverse camera systems by using a unified omnidirectional representation. It combines ray-distance Gaussian prediction, feature-space alignment, pose-free inference, panoramic adaptations, and an FoV-stratified benchmark.

  • Motivation: Monocular view synthesis remains ill-posed because a single image severely loses spatial information, despite advances in NeRF and 3D Gaussian Splatting.Novel view synthesis supports robotic navigation, AR/VR interaction, immersive telepresence, and 3D content creation.
  • Problem: Perspective-trained monocular 3DGS regressors such as SHARP and Flash3D fail to generalize to wide-FoV, fisheye, and panoramic cameras.Fine-tuning SHARP remains insufficient because its normalized-space mapping assumes a pinhole camera, while reprojection adds computational overhead and stitching artifacts.
  • Method: UniSHARP extends SHARP through a unified omnidirectional representation that organizes Gaussian primitives along rays and radial distances across camera systems.Its feature-space pipeline fuses 2D semantic encodings with 3D spatial features and predicts Gaussian residuals at native input resolution.
  • Method: UniSHARP supports pose-free monocular inference by using predicted ray fields to infer camera type and recover rendering geometry without manually provided intrinsics.For panoramic inputs, spherical Gaussian initialization and distortion-aware probabilistic dropout regularize Gaussian distributions under equirectangular distortion.
  • Evaluation: The FoV-stratified benchmark spans perspective, wide-FoV, fisheye, and panoramic cameras across real-world and simulated scenes, enabling analysis from 60◦ to 360◦.The benchmark includes established validation datasets, OmniRooms, and its projected wide-FoV variant; UniSHARP demonstrates state-of-the-art rendering quality and strong cross-camera generalization.

2 Related work

Related work spans multi-image, single-image, and wide-FoV novel view synthesis. UniSHARP builds on these directions by unifying perspective, wide-FoV, fisheye, and panoramic monocular synthesis in one feedforward 3DGS model.

  • Multi-image novel view synthesis: Multi-image NVS exploits cross-view consistency, progressing from neural radiance fields to explicit 3D Gaussian primitives with real-time rasterization.The passage also notes anti-aliased radiance-field variants for unbounded and high-resolution reconstruction.
  • Single-image novel view synthesis: Single-image NVS infers geometry, appearance, and visibility from priors, using neural radiance fields, multiplane images, layered geometry, inpainting, and adaptive MPI layouts.Larger reconstruction and synthesis models further demonstrate feedforward inference of plausible 3D structure from one image.
  • Wide-FoV novel view synthesis: Wide-FoV NVS must address projection distortion, nonuniform angular sampling, and boundary topology absent from perspective images, motivating ray-based representations and adapted camera models.Panoramic methods additionally use spherical radiance fields, spherical cost volumes, Gaussian pyramids, and Yin-Yang grids for 360-degree synthesis.
  • Wide-FoV novel view synthesis: Existing non-perspective methods commonly rely on optimized scenes, calibrated captures, or multiple panoramic observations, whereas UniSHARP targets unified monocular synthesis across camera systems.The stated camera systems are perspective, wide-FoV, fisheye, and panoramic settings within a single feedforward 3DGS model.

3 Method

UniSHARP enables universal monocular view synthesis by decoupling camera projection from scene representation in a unified ray-distance space. It combines geometry-anchored Gaussians, feature-conditioned residuals, mixed-camera training, panoramic adaptation, and pose-free camera recovery.

  • Unified ray-distance representation: UniSHARP represents each pixel with a predicted unit ray and radial distance, defining 3D points and Gaussian attributes consistently across heterogeneous camera models.The 3D point is xp = dprp, while placement, scale, and color are measured along rays rather than rasterized image-plane coordinates.
  • Gaussian scene construction: The method composes two-layer Geometry Anchored Gaussians with Feature Conditioned Gaussian residuals decoded from fused 2D semantic and 3D ray-based features.The second radial layer models disocclusions and high-frequency structures, while residual channels encode center offsets, inverse-distance, scale, quaternion, color, and opacity.
  • Mixed-camera training: UniSHARP jointly trains one model on perspective, wide-FoV, fisheye, and panoramic data without camera-specific branches.A weighted sampler draws source-target pairs from all supported datasets and groups each mini-batch by dataset for efficient collation and rendering.
  • Panoramic distortion adaptation: For equirectangular panoramas, training selectively drops the second Gaussian layer by latitude while always preserving the first layer for visible-surface coverage.This shifts panoramic distortion adaptation from a specialized prediction branch to a training-time allocation strategy.
  • Training objective: The objective combines appearance and depth supervision with Gaussian regularization, including inverse-distance smoothing, floater suppression, and multi-scale Sobel alignment.These terms stabilize weakly constrained Gaussian degrees of freedom and align rendered and supervised distance edges.
  • Pose-free deployment: UniSHARP also supports pose-free deployment by recovering camera geometry from the angular coverage of its predicted UniK3D ray field when calibration is unavailable.The calibrated setting remains supported when the input camera model and intrinsics are known.

4 Experiments

Experiments evaluate UniSHARP across perspective, wide-FoV, fisheye, and panoramic cameras using a unified FoV-stratified benchmark. Results show strong non-perspective performance, out-of-domain generalization, pose-free capability, and benefits from the proposed feature and Gaussian components.

  • Benchmark: The evaluation spans perspective, wide-FoV, fisheye, and panoramic monocular novel view synthesis under a unified FoV-stratified protocol.The benchmark diagnoses behavior from narrow perspective cameras to full 360° equirectangular inputs.
  • Zero-shot perspective evaluation: On Tanks and Temples, UniSHARP achieves the best PSNR and LPIPS, while SSIM remains close to Flash3D.This indicates preserved cross-dataset generalization with improved overall reconstruction fidelity.
  • Wide-FoV, fisheye, and panoramic performance: On OmniRooms-Wide and ScanNet++ Fisheye, UniSHARP improves over baselines across metrics and handles stronger projection distortion and larger angular coverage.It consistently improves PSNR, SSIM, and LPIPS on OmniRooms-Wide and outperforms PanoDreamer and Matrix3D on ScanNet++ Fisheye.
  • Pose-free performance: The pose-free variant maintains competitive rendering quality on WildRGB-D without camera calibration by estimating camera geometry from predicted rays.This demonstrates the practical value of ray-based camera recovery for unconstrained monocular inputs.
  • Model design: Ablations show that replacing learned 2D semantic and 3D geometric features with direct depth-RGB inputs causes the largest degradation on WildRGB-D and HM3D.Removing the second Gaussian layer also hurts performance, supporting its role in additional distance hypotheses for disocclusions and wide angular coverage.

5 Conclusion · A Additional Experiments and Ablations

UniSHARP is a universal-camera feedforward 3DGS framework for monocular novel view synthesis that reformulates Gaussian prediction in a shared ray-distance space. It combines Geometry Anchored Gaussians with Feature Conditioned Gaussian residuals while preserving single-image Gaussian regression efficiency.

  • 5 Conclusion: UniSHARP presents a universal-camera feedforward 3DGS framework for monocular novel view synthesis.
  • 5 Conclusion: The work addresses unreliable transfer from perspective-trained Gaussian regressors to heterogeneous camera systems.
  • 5 Conclusion: UniSHARP reformulates Gaussian prediction in a shared ray-distance space.
  • 5 Conclusion: The framework composes Geometry Anchored Gaussians with Feature Conditioned Gaussian residuals.
  • 5 Conclusion: This design preserves the efficiency of single-image Gaussian regression.
  • 5 Conclusion: The proposed framework supports heterogeneous camera systems through its universal-camera formulation.

A.1 Implementation Details

UniSHARP is trained on 8 H20 GPUs using a UniK3D ViT-L feature-only architecture with pretrained initialization. Training uses Adam for 106 iterations with warmup and cosine learning-rate decay, while wide-FoV and fisheye rendering uses the 3DGEER generic-camera Gaussian rasterizer.

  • A.1 Implementation Details: UniSHARP uses a feature-only architecture with a UniK3D ViT-L backbone initialized from pretrained weights.
  • A.1 Implementation Details: Experiments are conducted on 8 H20 GPUs, and wide-FoV and fisheye rendering uses the 3DGEER generic-camera Gaussian rasterizer.
  • A.1 Implementation Details: The model is optimized with Adam for 106 iterations, including a 104-iteration warmup followed by cosine learning-rate decay.
  • A.1 Implementation Details: The depth-head learning rate decays from 1.0 × 10−5 to 1.0 × 10−6.

A.2 Training Objective Ablation

The full training objective delivers the best rendering quality on both WildRGB-D and HM3D. Ablations show that rendered-depth supervision is most critical for PSNR, while perceptual appearance loss, TV regularization, and floater suppression improve fidelity and stability, especially panoramas.

  • A.2 Training Objective Ablation: The full objective achieves 21.56 PSNR and 0.143 LPIPS on WildRGB-D, and 29.24 PSNR and 0.065 LPIPS on HM3D.Each ablation removes one loss term from the full objective to measure its contribution to rendering quality.
  • A.2 Training Objective Ablation: Removing target rendered depth supervision causes the largest PSNR drop, reducing PSNR to 20.42 on WildRGB-D and 27.12 on HM3D.The ablation also increases LPIPS, although the WildRGB-D value is truncated in the supplied passage.
  • A.2 Training Objective Ablation: The perceptual appearance loss improves visual fidelity, while second-layer TV regularization and floater suppression stabilize the Gaussian field.These effects are reported qualitatively in the training-objective ablation analysis.
  • A.2 Training Objective Ablation: Removing floater suppression increases HM3D LPIPS from 0.065 to 0.153, highlighting its importance for panoramic scenes.The increase is attributed to unstable second-layer Gaussians near depth discontinuities.

A.3 Fisheye Dataset Visualization · A.4 Panoramic Inference via Cubemap Decomposition · A.5 Inference Time Comparison

The fisheye validation samples exhibit strong radial distortion and wide angular coverage, while panoramic cubemap inference causes stitching artifacts that UniSHARP avoids. UniSHARP also provides faster inference than the compared panoramic baselines through a single feedforward Gaussian prediction and direct rendering.

  • A.3 Fisheye Dataset Visualization: Fisheye validation samples show strong radial distortion and wide angular coverage, distinguishing native fisheye novel view synthesis from standard perspective evaluation.These samples are used in the benchmark’s fisheye validation data visualization.
  • A.4 Panoramic Inference via Cubemap Decomposition: SHARP cannot directly ingest equirectangular panoramas or other non-pinhole inputs because it maps normalized image pixels under a pinhole-camera assumption.A common workaround decomposes panoramas into six cubemap faces for independent SHARP processing.
  • A.4 Panoramic Inference via Cubemap Decomposition: Cubemap-based SHARP produces visible stitching artifacts at face junctions, whereas UniSHARP renders a coherent panoramic target view without seams.The comparison motivates avoiding pinhole-specific reprojection heuristics.
  • A.4 Panoramic Inference via Cubemap Decomposition: UniSHARP operates directly on panoramic inputs in unified ray-distance space, producing a seamless target instead of stitching independently inferred cubemap-face results.SHARP processes each cubemap face separately before stitching the rendered target views into equirectangular format.
  • A.5 Inference Time Comparison: 3.1 seconds: UniSHARP completes single-image novel view synthesis inference, compared with 8.6 seconds for PanoDreamer and 38.8 seconds for Matrix3D.Runtime is measured under the same evaluation setting, with relative runtime reported against UniSHARP.
  • A.5 Inference Time Comparison: 2.8× and 12.5× slower: PanoDreamer and Matrix3D, respectively, than UniSHARP.The speed advantage comes from predicting the complete Gaussian representation in one feedforward pass and directly rendering it, avoiding per-scene optimization.

B Limitations … D.6 Metric Implementation

UniSHARP’s benchmark evaluates universal monocular view synthesis across diverse camera systems using controlled local source-target pairs, standardized metadata, and reproducible metric aggregation. The paper also notes limitations in hallucinating large unseen regions and discusses broader applications and research value.

  • B Limitations: UniSHARP handles moderate disocclusions but may produce holes or weakly supported structures when target views reveal large regions outside the source image.As a feedforward Gaussian prediction model, it has limited evidence for hallucinating completely unseen content.
  • C Societal Impact: The unified perspective, wide-FoV, fisheye, and panoramic benchmark may support embodied AI, robotics, AR/VR, telepresence, spatial documentation, and geometry-aware spatial-intelligence research.The benchmark is intended to encourage systematic evaluation of camera-general view synthesis methods.
  • D Benchmark Details: The benchmark uses single-source multi-target samples with source images, target views, camera parameters when available, projection metadata, and effective FoV, while target RGB images remain unavailable during inference.Official splits are followed when available; otherwise, held-out scene- or sequence-level splits prevent leakage and fixed samples ensure deterministic evaluation.
  • D.1 Dataset Splits and Scene Selection: Dataset coverage preserves native or specified projections, including 130◦ equidistant wide-FoV and fisheye views, equirectangular panoramas, and perspective camera metadata.Perspective, wide-FoV, fisheye, and panoramic datasets include held-out or out-of-domain evaluation subsets as specified by their construction.
  • D.2 OmniRooms Construction: OmniRooms samples anchor cameras on a 0.5m voxel grid, retain centers with 60 ≤Z ≤180 cm, and generate 30 camera positions within a 30 cm local cube.Panoramas are rendered at 1024 × 2048 with aligned depth and metadata, while depth and geometry are withheld from model input at test time.
  • D.3 Source-Target Pair Filtering: Pairs are retained only when source-target overlap is at least 60%, camera-center distance is smaller than 0.5m, and image-index gap is at most 10.Overlap uses visible 3D points and circular horizontal wrap-around for panoramic data.
  • D.5 Evaluation Protocol: Evaluation renders each target in its camera projection and compares it with ground truth using PSNR, SSIM, and LPIPS after consistent resolution handling and valid-pixel masking.Panoramic rendering and filtering use longitude wrap-around, while metrics compare predictions and targets in the same equirectangular coordinates.
  • D.6 Metric Implementation: Metrics use normalized RGB values, RGB-channel SSIM with an 11-pixel Gaussian window and standard deviation 1.5, and LPIPS(net="alex").Scores are averaged across target views and source sequences, while camera-group averages are unweighted across dataset-level scores.

D.7 Baseline Evaluation Details … D.10 Licenses, Ethics, and Privacy

The supplementary sections detail standardized baseline and runtime evaluations, pose-free rendering protocols, quality-control filters, and dataset licensing and privacy practices. Together, these procedures define comparable evaluation conditions and document how released benchmark data are screened and governed.

  • D.7 Baseline Evaluation Details: All baselines use the same source-target pairs as UniSHARP, official checkpoints and code when available, compatible input representations, and an unchanged target-view protocol.Methods supporting only subsets of camera models use the closest compatible input representation.
  • D.7 Baseline Evaluation Details: Perspective baselines use original perspective camera parameters, official preprocessing, and target-resolution outputs for metric computation on perspective datasets.SHARP, Flash3D, LVSM, and TMPI are evaluated under corresponding pinhole cameras.
  • D.7 Baseline Evaluation Details: Wide-FoV, fisheye, and panoramic baselines convert benchmark inputs to official representations while preserving benchmark camera trajectories for target-view rendering or sampling.PanoDreamer and Matrix3D are evaluated on non-perspective datasets using official implementation settings.
  • D.7 Baseline Evaluation Details: Runtime comparisons fix hardware, rendered-view count, batch size, resolution, GPU, and iteration settings, while timing warm-up-complete forward passes and rendering but excluding loading and decoding.Measurements include model forward and target-view rendering.
  • D.8 Pose-Free Evaluation Details: The pose-free setting supplies only the source RGB image, allowing UniSHARP to predict a ray field, infer the camera model, and recover rendering geometry before synthesis.This evaluates operation without manually provided source-camera intrinsics.
  • D.8 Pose-Free Evaluation Details: On WildRGB-D, pose-free evaluation replaces source calibration with recovered camera geometry, while target camera parameters remain provided to define comparable evaluation views and metrics.The calibrated setting uses available camera parameters; pose-free rendering removes only source-camera calibration from model input.
  • D.9 Quality Control: Quality control removes samples with missing images, invalid camera metadata, invalid required depth, severe clipping, or invalid depth buffers, and applies valid fisheye and no-extrapolation masks.For simulated OmniRooms, retained camera centers and released local positions satisfy 60 ≤Z ≤180 cm.
  • D.10 Licenses, Ethics, and Privacy: Existing datasets follow their original licenses, while OmniRooms and OmniRooms-Wide use CC BY-NC 4.0 for research and non-commercial use and exclude private user information from released metadata.The benchmark combines public datasets with newly constructed OmniRooms and OmniRooms-Wide data.
Loading 2606.07514v1…