Source-linked AI summary
Complexity-guided Regularization for Generalizable Human Gaussian Splatting
Gun Ryu, Gi-Mun Um, Won-Sik Cheong, Wonjun Kim
TL;DR
Generalizable human Gaussian splatting still struggles to represent fine details, and prior methods have not sufficiently addressed regional Gaussian density. This method uses local geometric and textural complexity to guide density regularization and patch-based supervision, and reports consistently high-fidelity rendering. Its current scope is static multi-view reconstruction rather than dynamic video.
Problem
Generalizable human Gaussian splatting methods have not sufficiently addressed how densely Gaussians should be placed across regions, despite the complexity of fine-grained human details.
Source: §1 · p. 2Method
The method derives a local geometric-and-textural entropy prior to guide denser Gaussian placement in complex regions and sample complexity-weighted local patches for supervision.
Source: §1 · p. 2Results
The method consistently produces high-fidelity rendering results in generalizable human Gaussian splatting.
Source: §5 · p. 10Takeaways & Limitations
Complexity-guided density regularization and patch supervision provide guidance for reconstructing intricate human details.
Source: §5 · p. 10The method targets static multi-view reconstruction; video use would require computationally expensive per-frame reconstruction and may yield temporally inconsistent results.
Source: D Limitations and future directions · p. 21
Abstract
from arXiv · showhide
Recent studies on generalizable human Gaussian splatting have achieved the notable progress in synthesizing novel views of unseen human subjects from multi-view images. Despite this progress, existing methods still struggle to represent fine-grained details of human subjects, which inherently involve high geometric and textural complexity. This is because previous methods do not impose a constraint on how densely Gaussians are placed, which often leads to blurry rendering results in complex regions. To address this problem, we propose a complexity-guided regularization scheme. The key idea is to leverage a complexity prior computed via the local entropy in geometric and textural characteristics of a clothed human. This effectively prevents Gaussians from being coarsely placed in complex regions, which facilitates the faithful reconstruction of intricate structures. Furthermore, we sample local patches predominantly from complex regions according to this complexity prior, which allows the model to be sufficiently supervised by local details. Experimental results on benchmark datasets demonstrate that our method delivers state-of-the-art performance while maintaining the real-time inference speed in generalizable human Gaussian splatting. The code and model are publicly available at https://github.com/DCVL-3D/CGR-GS_release.
1 Introduction
Per-subject optimization limits applying human Gaussian splatting to unseen people, while generalizable methods have not sufficiently addressed how Gaussian density should vary across a subject. The proposed method uses local geometric and textural complexity to guide Gaussian placement and patch-based supervision toward fine details.
1 Introduction
- Earlier human 3D Gaussian splatting methods commonly require per-subject optimization, preventing direct application to unseen people without repeating training.Generalizable approaches instead regress Gaussian attributes for unseen subjects in a single forward pass.Source: §1 · p. 1
- Previous feed-forward methods place Gaussians by image pixels or skinned-mesh vertices, leaving their density insufficiently adapted to complex regions such as fingers or textured clothes.This can produce blurry renderings where fine-grained detail is under-represented.Source: §1 · p. 2
- Complex regions occupy little of a human-centric image, so whole-image supervision can let signals from simple regions dominate those from intricate details.The method therefore samples local patches with probability proportional to the complexity prior.Source: §1 · p. 2
- A per-vertex complexity prior, computed from local geometric and textural entropy, regularizes Gaussian density to be higher in complex regions.The goal is faithful reconstruction of fine-grained details.Source: §1 · p. 2
2 Related work
Prior human Gaussian splatting work includes per-subject optimization and generalizable feed-forward methods using view-pair geometry or skinned-body templates. This paper shifts the focus from estimating Gaussian positions to regulating their density across regions.
2 Related work
- Per-subject methods initialize or optimize Gaussian representations using human meshes or canonical poses, but their optimization requirement restricts generalization to unseen subjects.Some attach Gaussian points to mesh faces; others optimize attributes in a canonical pose and render arbitrary poses via forward skinning.Source: §2 · p. 3
- Generalizable methods regress attributes in one forward pass, using view-pair geometry or skinned human models as structural priors for Gaussian placement.View-pair approaches estimate per-pixel attributes; skinned-model approaches place Gaussians on the body surface or template.Source: §2 · p. 3
- Where earlier work focused on accurately estimating Gaussian positions, this method focuses on how densely Gaussians should be placed.The paper proposes regularization to improve generalizable human Gaussian splatting performance.Source: §2 · p. 3
3 Method
The method predicts human Gaussian attributes from multi-view inputs, then uses geometric and textural complexity to guide Gaussian density and local-patch supervision. Its training objective combines rendering, depth, and complexity-guided losses.
3.1 Architecture details
- Multi-view RGB images and Plücker ray maps are patch-tokenized, fully background patches are discarded, and a Gaussian decoder predicts attributes that are rasterized into target views.The predicted attributes include depth, spatial offset, scale, rotation, opacity, and color.Source: §3.1 · pp. 3–4
3.2 Complexity prior of the human subject
- The per-vertex complexity prior combines multi-scale geometric and textural entropy, assigning high values when either cue captures strong local variation.Geometric complexity measures variation in neighboring surface normals; textural complexity measures neighboring color deviations.Source: §3.2 · pp. 4–5
3.3 Complexity-guided regularization
- A pairwise ranking regularizer encourages local Gaussian density to increase with surface complexity, weighting pairs more strongly when their complexity gap is larger.Local density is estimated from distances to the k-nearest Gaussian centers; the regularizer penalizes a higher-complexity point having a lower density score.Source: §3.3 · pp. 5–6
3.4 Complexity-aware patch-based supervision
- Complexity-weighted patch sampling concentrates supervision on intricate regions, alternating local patches with full-image supervision and progressively shrinking patch size.Larger early patches provide broader structural context, while smaller later patches focus on fine-grained details.Source: §3.4 · p. 6
3.5 Loss function
- Training combines rendering, depth-consistency, and complexity-guided regularization losses, and applies the same objective during patch-based supervision.The rendering loss includes photometric, perceptual, and alpha-mask terms.Source: §3.5 · p. 6
4 Experiments
The experiments compare in-domain and cross-domain novel-view synthesis, then test the complexity-guided regularization and patch-sampling components through ablations. Across the reported benchmarks, the method improves reconstruction quality while retaining fast inference.
4.1 Implementation details
- Experiments use three multi-view datasets, trained independently for in-domain evaluation and with THuman2.0-only training for cross-domain tests; inference speed is reported as an average on THuman2.0.Evaluation uses PSNR, SSIM, LPIPS, and FPS; implementation uses 64 Gaussians per token and 2 patch centers per target image.Source: §4.1 · pp. 6–7, §4.2 · pp. 7–8
4.3 Performance evaluation
- In-domain, the method leads all datasets and metrics, with PSNR of 32.42 dB on THuman2.0, 31.68 dB on 2K2K, and 31.96 dB on X-Humans at 33.96 FPS; cross-domain PSNR gains are +0.93 dB and +0.96 dB.The cross-domain gains are over GHGS-MVSC on 2K2K and RoGSplat on X-Humans, respectively.Source: Table 1 · p. 7, §4.3 · p. 8, Table 2 · p. 8
- Qualitative comparisons show more accurate recovery of fine-grained fingers, facial details, and textured clothes than previous methods in both evaluation settings.The paper attributes prior methods’ degradation in these regions to insufficient Gaussian density.Source: Figure 4 · p. 7, §4.3 · pp. 8–9, Figure 5 · p. 8
4.4 Ablation study
- On THuman2.0, complexity regularization improves PSNR by 0.47 dB, while complexity-aware patch sampling improves PSNR by 0.43 dB over random sampling.Image-gradient sampling improves PSNR by 0.18 dB over random sampling; patch sampling concentrates supervision on intricate regions.Source: §4.4 · p. 9, Table 3 · p. 9, Table 5 · p. 9
- Performance improves as k increases from 3 to 7, with k = 7 best on PSNR and LPIPS; k = 9 remains comparable across metrics.The authors note that an excessively small k makes the density score noise-sensitive and choose k = 7 based on performance and computational cost.Source: §4.4 · p. 9, Table 4 · p. 9
- Combining geometric and textural complexity performs better than either alone, and maximum fusion outperforms average fusion by 0.11 dB in PSNR.The maximum operation preserves a strong cue when either geometric or textural variation is prominent.Source: §4.4 · p. 10, Table 6 · p. 10
- An image-derived complexity prior remains competitive without textured meshes, though its results are slightly lower, possibly because monocular-normal maps are inconsistent across views.With depth regularization, the variant reaches 32.04 dB PSNR, 3.21 dB above RoGSplat; without it, it reaches 31.83 dB, 0.90 dB above GHGS-MVSC.Source: §4.4 · p. 10, Table 8 · p. 10
5 Conclusion
The method allocates Gaussians according to local geometric and textural complexity and focuses patch-based supervision on complex regions. The authors report high-fidelity rendering and effectiveness in both in-domain and cross-domain settings.
5 Conclusion
- Complexity-guided Gaussian allocation and patch-based supervision concentrate representation and training signals on fine-scale regions, and the method produces high-fidelity results in in-domain and cross-domain settings.The complexity prior uses entropy of geometric and textural variations in clothed humans.Source: §5 · p. 10
Appendix — Complexity-guided Regularization for Generalizable Human Gaussian Splatting
- The appendix covers implementation details, additional ablations and comparisons, limitations and future directions, an ethics statement, and a supplementary video.Source: Appendix — Complexity-guided… · p. 14
A.1 Network architecture
The network embeds image patches as tokens, processes them with a transformer encoder, and decodes each token into attributes for 64 Gaussians.
A.1 Network architecture
- Each non-overlapping 8×8 patch becomes a 1024-dimensional token; a 24-layer, 1024-dimensional transformer with 64 heads feeds a decoder that predicts 15 attributes for each of 64 Gaussians per token.The attributes include depth, 3D spatial offset, scale, rotation quaternion, opacity, and color.Source: §A.1 · p. 14
A.2 Training details
Training precomputes the complexity prior and uses an AdamW schedule with warmup, cosine decay, and gradient clipping. Ablations without depth regularization retain gains from both proposed components, while the pairwise ranking formulation outperforms a Pearson-based alternative.
A.2 Training details
- Training runs for 800,000 iterations with batch size 2, taking approximately 3.5 days on one RTX 4090; the complexity prior is precomputed before training.AdamW uses 0.05 weight decay except for biases and normalization parameters, 2,000 iterations of linear warmup, cosine decay, and gradient clipping at 0.5.Source: §A.2 · p. 14
B.1 Effectiveness of the proposed components without depth regularization scheme
- Without depth regularization, complexity-guided regularization improves PSNR by 0.61 dB and patch-based supervision by 1.16 dB; together they improve PSNR by 2.21 dB over baseline.Source: §B.1 · p. 14, Table 10 · p. 15
B.2 Effectiveness of the pairwise monotonic ranking formulation
- The pairwise monotonic ranking formulation improves PSNR by 0.47 dB and SSIM by 0.0031 over baseline, whereas the Pearson formulation drops SSIM by 0.0015 and increases LPIPS by 0.0009.The ranking formulation preserves the order induced by complexity without requiring density scores to be linearly proportional to complexity.Source: §B.2 · p. 15, Table 11 · p. 15
B.3 Robustness to perturbations at training and inference phases
The method is tested under degraded training-time mesh and texture/geometry inputs and perturbed inference masks. The reported PSNR decrease is limited when both perturbations are applied together.
B.3 Robustness to perturbations at training and inference phases
- With 50% fewer mesh vertices and Gaussian noise σ = 0.02 on UV texture and geometry during training, or σ = 4-pixel mask jitter at inference, PSNR drops by 0.22 dB and 0.27 dB, respectively.Source: §B.3 · p. 15, Table 12 · p. 15
- Applying both training-time and inference-time perturbations together produces a 0.40 dB PSNR drop, which the authors describe as limited.Source: §B.3 · p. 15, Table 12 · p. 15
C.1 Analysis of the computational efficiency
The method maintains real-time inference while using less GPU memory than RoGSplat, and it leads the comparison across 3-, 4-, and 5-view settings. The reported quality advantage holds across all three input-view settings.
C.1 Analysis of the computational efficiency
- Over 26 FPS with 3,056 foreground tokens and up to 45.65 FPS with 1,437 tokens; 2.06 GB memory is below RoGSplat's 4.21 GB.The proposed method has 303.5 M parameters; RoGSplat has 12.3 M, while GHGS-MVSC has 987.2 M parameters and uses 10.47 GB.Source: §C.1 · p. 16, Table 14 · p. 16, Table 13 · p. 16
C.2 Comparison with different numbers of input views
- 2.43 dB, 1.49 dB, and 2.24 dB PSNR gains over GHGS-MVSC at 3-, 4-, and 5-view settings, respectively.The method achieves the best performance across all three metrics in each setting; Fig. 7 compares the reconstructions across input-view settings.Source: §C.2 · p. 16, Table 15 · p. 16, Figure 7 · p. 17
C.3 Comparison on the THuman2.1 dataset
On THuman2.1, the method performs best across all three metrics in both in-domain and cross-domain evaluation. Its PSNR advantage over the second-best method is larger in-domain than cross-domain.
C.3 Comparison on the THuman2.1 dataset
- 4.38 dB in-domain and 1.26 dB cross-domain PSNR gains over GHGS-MVSC on THuman2.1; the method leads across all three metrics in both settings.The cross-domain evaluation trains on THuman2.0 and evaluates directly on THuman2.1 without fine-tuning.Source: §C.3 · p. 17, Table 16 · p. 17
C.4 Comparison on real-world datasets
The method performs best on the reported real-world datasets and preserves face identity and geometry in face-region evaluation. It also outperforms LIFe-GoM in rendering quality while running faster under that comparison's protocol.
C.4 Comparison on real-world datasets
- 0.79 dB on DNA-Rendering and 1.33 dB on ZJU-MoCap PSNR gains over RoGSplat, with best performance across all three metrics on both datasets.Evaluation uses models trained on THuman2.0 without fine-tuning; Fig. 10 compares cross-domain results, and the prose reports clearer facial details and more accurate fingers.Source: §C.4 · pp. 17, 19, Figure 10 · p. 18, Table 17 · p. 19
C.5 Identity preservation on the face region
- 1.50 dB and 2.30 dB PSNR gains over GHGS-MVSC on THuman2.0 and X-Humans face regions, respectively, with best CSIM and AKD on both datasets.The evaluation also reports PSNR, SSIM, and LPIPS; Fig. 11 compares face-region reconstructions.Source: Table 18 · p. 19, §C.5 · p. 19, Figure 11 · p. 20
C.6 Comparison with LIFe-GoM
- 4.12 dB higher PSNR, 59.31 lower LPIPS*, and 7.7 FPS versus LIFe-GoM's 1.0 FPS under its evaluation protocol.The comparison uses 1024 × 1024 source and target resolutions and a 500 × 1000 crop centered on the subject.Source: Table 19 · p. 20, §C.6 · p. 20
C.7 Comparison with single-image human reconstruction methods
In comparisons with single-image human reconstruction methods, the method achieves better performance across all reported metrics while using four input views rather than one.
C.7 Comparison with single-image human reconstruction methods
- The method achieves better performance across all metrics than IDOL and LHM, using four input views while each comparator uses a single image.The comparisons follow each method's experimental setting: IDOL on THuman2.1 and LHM on 2K2K; reported results differ from Tables 1 and 16 because evaluation protocols differ.Source: §C.7 · p. 20, Table 20 · p. 21
D Limitations and future directions
The authors identify static-only reconstruction and dependence on accurate clothed human meshes as limitations. They also discuss possible misuse and report that their experiments omit error bars.
D Limitations and future directions
- The method handles static multi-view images, while video use would require per-frame reconstruction that may be expensive and temporally inconsistent.The authors suggest incorporating a parametric human body model and linear blend skinning as future work for temporal poses.Source: D Limitations and future directions · p. 21
- Training requires an accurate clothed human mesh, which is challenging to obtain and typically depends on labor-intensive 3D scanning.The authors propose computing complexity priors directly from multi-view images to reduce this dependency.Source: D Limitations and future directions · p. 21
E Ethics statement
- The authors caution that realistic human reconstruction may raise privacy, identity-misuse, and unauthorized-likeness concerns, and state that the method is intended solely for research use.They encourage responsible deployment and adherence to ethical guidelines.Source: E Ethics statement · p. 21
NeurIPS Paper Checklist
- The experiments report no error bars; the authors say results come from standard randomly seeded runs without cherry-picking.They describe these runs as a fair and representative evaluation.Source: NeurIPS Paper Checklist · p. 24