Source-linked AI summary
GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis
Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, Yebin Liu
TL;DR
Sparse-view human novel view synthesis needs photo-realistic rendering without the per-subject optimization used by many prior representations. GPS-Gaussian regresses pixel-wise Gaussian parameters from source views and jointly estimates depth to lift them into 3D. It reports state-of-the-art comparisons, 2K rendering exceeding 25 FPS, and no fine-tuning or optimization for unseen characters, while requiring accurate foreground matting preprocessing.
Problem
Sparse-view human NVS remains difficult because prior approaches rely on dense views, precise geometry, or per-subject optimization, limiting instant rendering of unseen subjects.
Method
GPS-Gaussian uses 2D source-view Gaussian parameter maps, an iterative depth estimator, and jointly trained Gaussian regression to construct 3D Gaussians in a feed-forward manner.
Results
GPS-Gaussian outperforms the compared methods on all reported metrics and synthesizes 2K-resolution novel views exceeding 25 FPS on a single modern graphics card.
Takeaways & Limitations
The method enables instantaneous novel-view rendering of unseen characters without fine-tuning or optimization under sparse-view camera settings.
Takeaways & Limitations
Accurate foreground matting is necessary as a preprocessing step for the method’s human novel-view synthesis.
Abstract
from arXiv · showhide
We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner. The proposed method enables 2K-resolution rendering under a sparse-view camera setting. Unlike the original Gaussian Splatting or neural implicit rendering methods that necessitate per-subject optimizations, we introduce Gaussian parameter maps defined on the source views and regress directly Gaussian Splatting properties for instant novel view synthesis without any fine-tuning or optimization. To this end, we train our Gaussian parameter regression module on a large amount of human scan data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. The proposed framework is fully differentiable and experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.
1. Introduction
Human novel view synthesis seeks photo-realistic, real-time rendering from sparse camera views, where prior implicit, image-based, and point-based methods face optimization, geometry, or efficiency limitations. GPS-Gaussian addresses this with feed-forward pixel-wise Gaussian regression, learned depth lifting, and joint training.
- Motivation: Sparse-view human NVS remains challenging because high-fidelity rendering requires handling limited views, self-occlusion, and appearance consistency.The task supports applications including sports broadcasting, stage performance, and holographic communication.
- Prior limitations: 3D Gaussian Splatting offers explicit, differentiable, and highly efficient rendering through Gaussian primitives with learnable position, color, opacity, and covariance.Its alpha-blending mechanism supports gradient-based optimization and real-time rendering.
- GPS-Gaussian: GPS-Gaussian directly regresses Gaussian parameters from source views using 2D pixel-wise maps instead of per-subject optimization or unstructured point clouds.The maps enable efficient 2D convolution networks and represent each foreground pixel as a Gaussian point.
- GPS-Gaussian: Iterative stereo depth estimation jointly trains with Gaussian regression to lift 2D maps into 3D while improving position accuracy and reducing rendering artifacts.The framework is fully differentiable and achieves 2K rendering exceeding 25 FPS on a single modern graphics card.
- Prior limitations: Implicit human representations can produce high quality but typically process queried points through full networks and often require per-subject optimization.Generalization to unseen subjects is non-trivial for such methods.
- Prior limitations: Image-based rendering commonly depends on dense input views or precise geometry proxies, limiting its suitability for sparse-view settings.High-quality multi-view stereo and surface reconstruction are required to obtain useful proxies.
3. Preliminary
3D Gaussian Splatting represents scenes with parameterized Gaussian primitives, projects them into image space, and renders them through alpha blending. GPS-Gaussian applies this representation to source-view parameter maps for novel-view rendering.
- 3D Gaussian representation: The covariance matrix is decomposed into scaling and rotation matrices so Gaussian parameters can be optimized by gradient descent.This decomposition provides an optimization-friendly parameterization of anisotropic Gaussian shape.
- Projection: Projection maps each 3D Gaussian into the image plane using a view transformation and the Jacobian of an affine projective approximation.The projected covariance is computed in 2D image space.
- Rendering: Rendered pixel colors are produced by point-based alpha blending over projected Gaussians.Each point contributes according to its color and a density formed from a 2D Gaussian and learned opacity.
- 3D Gaussian representation: Each 3D Gaussian is parameterized by a position, color, rotation, scaling factor, opacity, and covariance-related properties.The representation explicitly models point primitives rather than an implicit scene field.
- GPS-Gaussian pipeline: GPS-Gaussian selects adjacent source views, estimates depth, forms Gaussian representations from RGB and depth, and aggregates unprojected points for novel-view rendering.Its fully differentiable design supports joint training of the network components.
4. Method
GPS-Gaussian selects neighboring source views, estimates depth, predicts pixel-wise Gaussian parameters, and lifts them into 3D for novel-view rendering. Joint differentiable training links depth accuracy with Gaussian rendering quality.
- View Selection: The method selects the two source views nearest to the target viewpoint as a binocular-stereo working set.View proximity is determined by dot products between input-view and target-view direction vectors.
- Depth Estimation: A shared image encoder extracts dense features from rectified source images, while iterative stereo matching predicts depth maps for both views.The depth module avoids prohibitively slow 3D convolutions and uses camera parameters to convert disparity into depth.
- Gaussian Parameter Prediction: Pixel-wise Gaussian maps predict position, color, rotation, scaling, and opacity for each foreground image pixel.Depth-aware features are fused with image features in a U-Net-like decoder to regress full-resolution Gaussian parameters.
- 3D Lifting and Rendering: Predicted depth unprojects Gaussian parameter maps from both source views into 3D, after which Gaussian Splatting renders the target view.The learnable unprojection bridges 2D image planes and the 3D Gaussian representation.
- Joint Training: Differentiable rendering jointly trains both source-view depth estimation and Gaussian parameter prediction using image and depth supervision.The rendering objective combines L1 and SSIM image losses, while depth predictions receive weighted sequence supervision.
- Evaluation Setting: The evaluation compares feed-forward generalizable methods with optimized 3D-GS using two 1024 × 1024 source images on an RTX 3090 GPU.GPS-Gaussian and FloRen use TensorRT for fast inference, whereas 3D-GS requires per-subject optimization.
5. Experiments
GPS-Gaussian is evaluated on synthetic and real-world human data under sparse-view settings, with comparisons, camera-sparsity tests, and ablations of its training and depth components.
- Comparisons with state-of-the-art methods: GPS-Gaussian outperforms ENeRF, FloRen, IBRNet, and optimized 3D-GS on all reported metrics while rendering substantially faster.The baselines use two source views, whereas original 3D-GS is optimized on all eight input views.
- Comparisons with state-of-the-art methods: Qualitative comparisons show more detailed human appearances and more reasonable geometry on THuman2.0, Twindom, and collected real-world data.The comparison covers both synthetic datasets and real-world captures.
- Camera sparsity: At 6 cameras, the model trained with 8 cameras degrades reasonably and remains robust without fine-tuning.Original 3D-GS is omitted because per-subject optimization takes several minutes and produces noisy results even with eight cameras.
- Ablation studies: Joint training improves depth estimation by 5% in EPE and reduces rendering noise caused by depth ambiguity in source-view margins.Gaussian rendering loss helps rectify depth artifacts, while predicted low opacity suppresses problematic points.
- Ablation studies: Removing the depth encoder produces blurred renderings because image features alone inadequately regress Gaussian scaling maps.Scaling depends on depth, texture, and surface roughness, so removing spatial awareness degrades LPIPS.
6. Discussion
GPS-Gaussian directly regresses pixel-wise Gaussian parameter maps for real-time, photo-realistic human novel-view synthesis under sparse camera settings, while retaining important scope and data requirements.
- Conclusion: The method improves quantitative and qualitative results over baselines and renders much faster on a single RTX 3090 GPU.It targets real-time human novel-view synthesis under sparse-view camera settings.
- Limitations: Accurate foreground matting is required because the method focuses on synthesizing novel views of human performers.This makes straightforward generalization to more general tasks difficult.
- Limitations: Ground-truth depths are required for supervision, increasing the difficulty of acquiring training data.The paper suggests massive high-quality synthetic data covering varied scenarios as a possible way to alleviate these issues.
Supplementary Material
The supplementary material documents opacity and scaling visualizations, camera-setup generalization, runtime, architecture, and live-demo settings.
- Supplementary contents: Supplementary sections cover opacity maps, scaling maps, randomly placed cameras, runtime comparison, network architecture, and live-demo settings.These sections extend the main experimental and implementation description.
7. Visualization of Opacity Maps
The opacity-map visualization explains how learned opacity suppresses ambiguous or outlier Gaussian points during novel-view rendering.
- Figure 5: Low predicted opacity makes outlier points invisible in novel views, reducing noise from ambiguous depth near occlusion boundaries.Examples include overlapping legs and crossed arms whose low-resolution depth estimates change sharply.
- Figure 5: Figure 5 maps source images, predicted opacity, projected color and opacity, and novel-view renderings across panels (a)–(e).Cold colors indicate opacity near 0, while hot colors indicate values near 1.
- Figure 6: Figure 6 visualizes source imagery, depth, normalized scaling, and zoomed Gaussian-point shapes, with hotter colors denoting larger scaling values.The depth and scaling maps are normalized for visualization.
8. Visualization of Scaling Maps
The scaling map shows that Gaussian scale generally increases with depth, while fine-grained textures and high-frequency geometry produce smaller Gaussians.
- Gaussian points at lower depth generally have smaller scales than more distant points.
- Fine-grained textures and high-frequency geometries lead to small-scaled Gaussians.
9. Randomly Placed Camera Setup
GPS-Gaussian generalizes to randomly placed cameras with substantial pitch and yaw variation, although broader synthetic camera coverage can further improve performance.
- The model generalizes strongly to random camera setups with pitch from [−20◦, +20◦] and yaw from [−25◦, +25◦].The result is illustrated using extreme-pitch and extreme-yaw source views, a novel rendering, and ground truth.
- Retraining with synthetic data covering more general camera setups is recommended for improved performance in such cases.
10. Run-time Comparison
GPS-Gaussian shifts most computation to source-view processing and renders each requested novel view efficiently, supporting concurrent real-time rendering.
- 0.8 ms is required to render the 3D Gaussians to a desired novel view after source-view computation.GPS-Gaussian’s source-view computation includes binocular depth estimation and Gaussian parameter map regression.
- Unlike several baselines, GPS-Gaussian avoids recomputing novel-view-correlated modules when the target viewpoint changes.The baselines recompute viewpoint-dependent modules such as cost volumes, depth prediction, or sampling.
- GPS-Gaussian can render multiple novel views concurrently in real time from two 1024 × 1024 source images.The runtime comparison is reported on an RTX 3090 GPU.
11. Network Architecture
GPS-Gaussian uses an image encoder, depth estimator, and Gaussian parameter predictor to transform source-view images into renderable Gaussian representations.
- The network architecture comprises an image encoder, depth estimator, and Gaussian parameter predictor.
- The image encoder extracts dense multi-scale features from both source images for correlation-volume construction and Gaussian parameter regression.Its feature maps are produced at 1/2, 1/4, and 1/8 input resolutions with 32, 48, and 96 channels.
- The depth estimator re-indexes correlation volumes for both lookup procedures, enabling a compact and parallelized implementation.
- The system synthesizes novel views from live RGB streams captured by six uniformly positioned cameras on a 2-meter-radius circle.The setup uses an RTX 3090 GPU and supports challenging hairstyles and human-object or multi-human interactions.