Source-linked AI summary
Sharp Monocular View Synthesis in Less Than a Second
Lars Mescheder, Wei Dong, Shiwei Li, Xuyang Bai, Marcel Santos, Peiyun Hu, Bruno Lecouat, Mingmin Zhen, Amaël Delaunoy, Tian Fang, Yanghai Tsin, Stephan R. Richter, Vladlen Koltun
TL;DR
Single-image view synthesis must recover a useful 3D scene representation quickly enough for interactive nearby-view rendering. SHARP predicts a metric 3D Gaussian representation in one feedforward pass, achieving state-of-the-art zero-shot fidelity while substantially reducing synthesis time. Its main scope is fast, high-resolution synthesis for nearby views, with faraway-view synthesis left as future work.
Problem
Single-image view synthesis seeks fast, photorealistic nearby views from one photograph, with metric scale for coupling virtual and physical cameras.
Method
SHARP uses a single neural-network forward pass to regress a high-resolution metric 3D Gaussian representation from one photograph.
Results
SHARP reduces LPIPS by 25–34% and DISTS by 21–43% versus Ren et al. (2025) across zero-shot test datasets while accelerating synthesis by three orders of magnitude.
Takeaways & Limitations
SHARP demonstrates state-of-the-art high-resolution nearby-view synthesis with under-one-second generation and real-time rendering from a single image.
Takeaways & Limitations
The current scope emphasizes nearby views; extending photorealistic synthesis to faraway views without sacrificing nearby-view fidelity and fast interaction remains future work.
Abstract
from arXiv · showhide
We present SHARP, an approach to photorealistic view synthesis from a single image. Given a single photograph, SHARP regresses the parameters of a 3D Gaussian representation of the depicted scene. This is done in less than a second on a standard GPU via a single feedforward pass through a neural network. The 3D Gaussian representation produced by SHARP can then be rendered in real time, yielding high-resolution photorealistic images for nearby views. The representation is metric, with absolute scale, supporting metric camera movements. Experimental results demonstrate that SHARP delivers robust zero-shot generalization across datasets. It sets a new state of the art on multiple datasets, reducing LPIPS by 25-34% and DISTS by 21-43% versus the best prior model, while lowering the synthesis time by three orders of magnitude. Code and weights are provided at https://github.com/apple/ml-sharp
1 Introduction
SHARP targets fast, metric, nearby-view synthesis from a single photograph by directly predicting a high-resolution 3D Gaussian representation. Its design combines an end-to-end network, artifact-suppressing losses, and depth alignment, achieving strong zero-shot fidelity with rapid synthesis.
- Problem setting: The application targets interactive nearby-view browsing, real-time photorealistic rendering, and stable AR/VR posture shifts rather than substantial travel through the scene.The desired representation must preserve absolute metric scale.
- Approach: SHARP produces a metric 3D Gaussian representation from one photograph through a single neural-network forward pass in under one second on an A100 GPU.The metric representation provides absolute scale for coupling virtual cameras with physical devices.
- Design: The method uses an end-to-end architecture, a loss configuration prioritizing view fidelity while suppressing artifacts, and a learned depth-adjustment module for ambiguous depth supervision.These components are validated through controlled experiments.
- Experimental outcome: 25–34% lower LPIPS and 21–43% lower DISTS than Ren et al. (2025) across zero-shot test datasets, with three-orders-of-magnitude faster synthesis.SHARP also supports high-resolution nearby-view rendering at 100 frames per second.
- Conclusion: The paper demonstrates that state-of-the-art high-resolution view synthesis is feasible with a purely regression-based framework.This conclusion follows the proposed architecture and experimental evaluation on datasets not used during training.
2 Related Work
Prior work spans image-based rendering and neural representations for synthesizing views from multiple or sparse images, alongside single-image pipelines using warping, multiplane images, radiance fields, and Gaussian prediction. Diffusion methods broaden view synthesis toward high-quality faraway views from sparse or single images.
- View synthesis from multiple images: Multi-image view synthesis evolved from image interpolation and layered depth representations to neural radiance fields and 3D Gaussian Splatting.These approaches generally rely on multiple captured viewpoints or scene information.
- Sparse-view feedforward synthesis: Feedforward methods predict view-synthesis representations from a small number of nearby views using learned features, cost volumes, radiance fields, or transformers.Examples include IBRNet, MVSNeRF, LaRa, and GS-LRM.
- View synthesis from a single image: Single-image view synthesis uses learning-based pipelines to infer plausible 3D structure through appearance flow, depth warping, multiplane images, radiance fields, and per-pixel Gaussian prediction.Later methods incorporate pretrained depth prediction or decompose the task into multiple stages.
- Diffusion-based synthesis: Diffusion-based systems use iterative denoising and have been applied to novel-view synthesis from sparse image sets or a single image.Scene-level systems leverage diffusion priors to support impressive image quality from faraway viewpoints.
3 Method
SHARP predicts a metric, high-resolution 3D Gaussian scene representation from one image in a single forward pass, using depth adjustment and attribute refinement before real-time rendering.
- 3.1 Overview: SHARP maps one 1536 × 1536 RGB image to approximately 1.2 million 3D Gaussians representing position, scale, orientation, color, and opacity.The representation uses two layers on a 768 × 768 grid and omits spherical harmonics to control output size.
- 3.1 Overview: The architecture combines a pretrained feature encoder, two-layer depth decoder, depth adjustment module, Gaussian initializer, Gaussian decoder, composer, and differentiable renderer.The predicted Gaussians are rendered to input and novel views for training supervision.
- 3.1 Overview: Depth adjustment addresses monocular depth ambiguity by learning a scale map that modifies the predicted two-layer depth used to initialize Gaussians.The adjustment is motivated by artifacts caused when ambiguous monocular depth estimates are used for view synthesis.
- 3.1 Overview: The Gaussian decoder predicts refinements for every Gaussian attribute, while the composer converts base Gaussians and refinements into final attributes through attribute-specific activations.Refining position, scale, rotation, color, and opacity supports detailed geometry and appearance.
- 3.1 Overview: SHARP renders the synthesized representation from arbitrary viewpoints using camera projection parameters, with inference amortized so new views render in real time.The projection incorporates source and target camera parameters when Gaussians are predicted in normalized space.
- 3.1 Overview: The model is trained end-to-end and produces approximately 1.2 million Gaussians in under one second on a single GPU.Its modules include a Depth Pro-based encoder and DPT-based depth and Gaussian decoders.
4 Experiments
SHARP is evaluated zero-shot across metric-pose datasets against feedforward, Gaussian, multiplane, regression, and diffusion baselines, with quantitative, qualitative, and ablation analyses.
- 4 Experiments: The evaluation covers Middlebury, Booster, ScanNet++, WildRGBD, ETH3D, and Tanks and Temples, using datasets with metric poses.Non-metric datasets such as RealEstate10K are excluded.
- 4 Experiments: LPIPS and DISTS are the primary evaluation metrics, and all methods are assessed at full ground-truth resolution after resizing lower-resolution outputs when necessary.The setup emphasizes perceptual fidelity because pointwise metrics can be overly sensitive to small translations.
- 4 Experiments: SHARP is compared with 3D Gaussian, multiplane-image, image-regression, and diffusion-based state-of-the-art methods.The baselines include Flash3D, TMPI, LVSM, SVC, ViewCrafter, and Gen3C.
- 4 Experiments: SHARP achieves the highest accuracy on all reported metrics across all datasets in the zero-shot evaluation.Table 1 reports mean metric values over all test samples, with lower values better.
- 4 Experiments: Qualitatively, SHARP consistently produces higher-fidelity renderings from nearby views than the displayed baselines.Additional qualitative results cover images from all evaluation datasets.
- 4 Experiments: Ablations find that perceptual loss improves visual quality, regularizers address artifact classes, learned depth adjustment sharpens details, and SSFT produces crisper views.The study varies losses, training curriculum, depth adjustment, and related components.
5 Conclusion
SHARP enables real-time, high-resolution nearby-view synthesis from a single photograph by predicting a 3D Gaussian representation in under a second. The method achieves state-of-the-art fidelity while leaving faraway-view synthesis as an open direction.
- SHARP predicts a 3D Gaussian representation from one photograph in under a second, then renders high-resolution nearby views in real time.
- SHARP achieves state-of-the-art image fidelity for nearby-view synthesis while outperforming methods that can require two to three orders of magnitude more computation.
- Extending SHARP to photorealistic faraway views without sacrificing nearby-view fidelity or fast interactive synthesis remains a future-work objective.
Supplementary Material
The supplementary methods describe output parameterization, metric-aware position processing, loss weights, and view-frustum masking for reliable supervision. The mask excludes target-view regions whose correspondence to the source view is unreliable.
- The supplementary material specifies activation functions, scale factors, and weighted loss coefficients used in the training configuration.
- SHARP predicts Gaussian position, scale, color, rotation, and opacity attributes, with position activations applied in normalized device coordinates before returning to world coordinates.
- View-frustum masking uses depth and reprojection to supervise only target-view regions corresponding to points visible in the source view.
A.4 The Perceptual Loss
The perceptual-loss design improves rendered detail but introduces substantial memory and sharpness challenges. SHARP addresses memory through computation-graph surgery and sharpness through a Gram-matrix loss, with reported gains in DISTS and rendering latency.
- Perceptual loss on full images can cause out-of-memory conditions because ResNet feature computation accumulates a large backpropagation graph.
- Computation-graph surgery caches and injects gradients, releasing the ResNet graph while preserving full-FP32 perceptual-loss training on reconstruction and novel views.
- Latent-space perceptual loss can produce blurry renderings and large, blobby Gaussians, so a Gram-matrix loss is used to encourage sharper images.
- The tuned perceptual-loss configuration improves DISTS by 62% and 47% and reduces rendering latency by 49% and 36% on the reported benchmarks.
B.1 Synthetic data
SHARP’s synthetic training data combines artist-made indoor and outdoor environments with procedural variation, photorealistic rendering, and multi-camera captures. A second stage adds OpenScene and licensed real photographs at large scale.
- Stage 1 samples from over 2K outdoor and 5K indoor artist-made scenes and procedurally augments their content and appearance.
- Each scene uses a ring of 10 virtual cameras placed within 60 cm of one another to capture nearby perspectives with V-Ray rendering.
- Stage 2 combines OpenScene with licensed photographs from Shutterstock, Getty Images, and Flickr, totaling 2.65M images.
C.1 Image Fidelity Metrics
The evaluation favors perceptual metrics because PSNR and SSIM can mistake small spatial shifts for severe errors, whereas DISTS and LPIPS better reflect perceptual similarity.
- The metric experiment compares responses to translated, mean, and depth-related image variations to identify suitable view-synthesis fidelity measures.The supplied passages motivate perceptual evaluation but do not report a separate quantitative result for the depth-ambiguity example.
- 1% translation reduces PSNR to 11.2 and SSIM to 0.375, nearly matching a mean image at PSNR 10.7 and SSIM 0.351.These pointwise metrics therefore cannot reliably distinguish a near-identical shifted image from an unrecognizable mean image.
- DISTS remains 0.079 for a 1% translation versus 0.859 for a mean image, demonstrating greater robustness to small shifts.This robustness is relevant because geometric inaccuracies in view synthesis can appear as small image displacements.
D.1 Evaluation Dataset Setup
The evaluation uses paired views from multiple real-world datasets, selected by depth overlap and bounded in number because diffusion baselines are slow to evaluate.
- The setup spans ScanNet++, WildRGBD, Tanks and Temples, and ETH3D, with stereo evaluation predicting the right frame from the left frame.Dataset-specific sampling uses the ScanNet++ test split, WildRGBD validation split, and metric-space depth construction for Tanks and Temples and ETH3D.
- Evaluation pairs are selected from 10-view scene sets using pairwise depth overlap above 60%, with at most 512 pairs per dataset.For each selected pair, the target image is predicted from the source image.
- The 512-pair cap is driven by diffusion runtime: Gen3C takes 15 minutes per synthesized view, making evaluation of 512 pairs roughly five days on an A100.
- The study also reports model parameter counts and retrains Flash3D on shared synthetic data to test whether data quality explains SHARP’s fidelity.Flash3D shows no distinct improvement with the in-house synthetic data, indicating that this data is not the principal factor in the reported performance.
D.3 Additional Quantitative Experiments
Additional experiments report runtime and privileged-depth evaluations while cautioning that PSNR and SSIM are unsuitable primary measures of view-synthesis fidelity.
- PSNR and SSIM are reported for completeness but discouraged for view-synthesis fidelity evaluation because of their sensitivity to small translations.
- SHARP synthesizes its 3D representation in less than one second on an A100 and renders it at 100 FPS or higher on most datasets.Runtime varies with native dataset resolution, including ETH3D’s 6048 × 4032 images.
- Privileged ground-truth depth is used to evaluate view-synthesis fidelity after scale adjustment, with proxy-depth methods aligned to ground truth or other methods assessed through a DISTS-based scale sweep.The experiment compares methods under privileged depth information rather than the standard monocular-depth setting.
- The privileged-depth setup uses method-specific depth proxies, including UniDepth, Dust3r, DepthPro, MoGe, and DepthPro for SHARP.
D.5 Ablation Studies
Ablations show that SHARP’s loss design, learned depth adjustment, backbone training, and Gaussian count affect fidelity, while its strongest operating regime is nearby-view synthesis.
- Losses: Depth loss reduces geometry distortion, perceptual loss improves inpainting quality and sharpness, and regularizers improve difficult scenes and rendering speed despite limited metric changes.The Gram-matrix component also significantly improves results.
- Depth Adjustment: Learned depth adjustment consistently improves perceptual fidelity and produces sharper synthesized views.
- Self-supervised Fine-tuning: Self-supervised fine-tuning yields sharper qualitative images but does not consistently improve metrics across the ablation datasets.The authors hypothesize that synthetic data contains few complex view-dependent effects.
- Unfreezing Backbone: Unfreezing the monodepth backbone improves fidelity quantitatively and qualitatively, including fewer boundary and reflection artifacts.
- Gaussian Count: Increasing the predicted Gaussian count improves performance, while the full output contains approximately 1.2 million Gaussians.
- Limitations: SHARP is not designed for faraway views with little source-image overlap and can fail on challenging depth cases such as depth-of-field, starry skies, and reflections.The authors attribute these failures to corrupted depth initialization and identify them as a long-tail depth-prediction problem.
- D.6 Motion Range: SHARP performs best for camera baselines below 0.5 meters and remains best or second best up to 3 meters, with diffusion methods gaining advantages on some larger motions.Gen3C outperforms SHARP only beyond 3 meters on ETH3D and beyond 0.5 meters on ScanNet++ in the cited setup.