Source-linked AI summary
GradRig: Differentiable Weights for Skinned Gaussian Splat Deformation
Nina Vesseron, Élie Michel
TL;DR
Gaussian splats lack connectivity, so standard skinned deformation can create holes when neighboring splats undergo different transformations. GradRig uses skinning-weight gradients for mesh-free, real-time deformation and adds optional adaptive resampling to mitigate affine-approximation artifacts. Experiments report more accurate gradients than nearest-neighbor estimation and improved visual quality after resampling.
Problem
Gaussian splats have no explicit connectivity, limiting standard skinned deformation when neighboring splats follow different transformations.
Method
GradRig estimates deformation-field variation from skinning-weight gradients derived during authoring and optionally adaptively resamples splats in rest space.
Results
Automatic authoring-time gradient derivation is more accurate than Moving Least Squares on 16 nearest neighbors, while resampling significantly improves visual quality in rapidly varying deformation fields.
Takeaways & Limitations
The method provides mesh-free interactive Gaussian-splat deformation without scene-specific optimization or mesh extraction, with resampling available to reduce artifacts.
Takeaways & Limitations
Experiments limit each splat to at most two influencing rig nodes, affecting 8.5% of Gaussian splats in one evaluated real scenario; the formulation is demonstrated for linear blend skinning.
Abstract
from arXiv · showhide
Skinned deformation is a common framework to turn a 3D shape from its rest pose into a dynamic pose through the deformation of a coarser kinematic structure, called rig. When applied to a 3D mesh, this rig only needs to displace vertices to deform the polygons that connect them. However, when deforming 3D Gaussian Splats, which do not provide connectivity information, rigidly transforming points is not enough to prevent the creation of holes when stretching shapes. In this paper, we use the spatial gradient of skinning weights to provide a full mesh-free deformation pipeline for Gaussian Splats, that more accurately stretches splats while remaining fully compatible with real-time rendering capabilities, which we demonstrate in a WebGL viewer. We present how we evaluate these gradients when the user creates the rig structure and propose an optional adaptive resampling scheme to split up splats that still produce artifacts.
1 Introduction
The paper adapts rigged deformation to mesh-free Gaussian splats, where missing connectivity makes independent point transforms produce holes or overlaps. It uses spatial skinning-weight gradients to estimate neighbor deformation, with adaptive resampling for artifacts, while targeting GPU-friendly real-time manipulation without mesh extraction or scene-specific optimization.
- Motivation: Gaussian splats lack mesh connectivity, so independently deforming their elements can create holes or overlap when neighboring points undergo different transformations.Meshes deform connected polygons through shared vertices, whereas splats carry separate surface or volume elements without explicit neighbors.
- Method: The method estimates neighboring deformation with the spatial gradient of skinning weights, avoiding explicit neighbor lookups and keeping the computation easy to parallelize on a GPU.The overall deformation gradient is obtained from gradients evaluated at splat centers and stored as additional splat attributes.
- Scope and capability: The system enables real-time manipulation on consumer-grade devices without mesh extraction, scene-specific optimization, or access to the original images.The paper presents the approach as applicable to pre-existing Gaussian splat scenes and compatible with real-time rendering.
- Method: Skinning-weight gradients are derived during authoring from the user’s selection operations, avoiding discretization artifacts without changing the authoring workflow.The framework models weight construction through sequences of selections that add, subtract, or multiply weights.
- Artifact mitigation: Adaptive resampling splits splats whose affine deformation approximation produces artifacts, enforcing a bound on the second-order Taylor-expansion term in highly deformed regions.The resampling is performed in rest space and can therefore be reused across similar or less intense deformations.
4 Results
The method reduces deformation artifacts through differentiable skinning gradients and optional adaptive resampling, while preserving interactive rendering and manipulation across large Gaussian-splat scenes.
- Implementation: The Web-based renderer stores two node indices, weights, and gradients per splat, adding 10 bytes per splat with 8-bit encoding.The sparse representation limits each splat to two non-zero weights.
- Quality: The method substantially reduces artifacts from regular skinned deformation across translations and torsions, while reproducing deformations at least as large as Real-time Large-scale Deformation.The comparison is qualitative because ground truth for deformed objects is unavailable.
- Performance: The added per-splat transform has no significant rendering-speed impact on an Apple M1 Max across scenes containing 85K–850K splats.Rendering is fragment-bound, and viewpoint changes affect speed more strongly than the deformation process.
- Quality: The differentiable skinning gradients produce more accurate deformation than 16-neighbor Moving Least Squares estimates, especially where gradients vary rapidly.The ablation reports fewer visual artifacts and shows that KNN estimation cannot match the differentiable approach in rapidly varying regions.
- Performance: Gradient computation makes weight editing about 3 times slower but remains interactive and is about 10 times faster than KNN estimation.This overhead is paid during one-time skinning setup rather than deformation; the supplied passage truncates the comparison context after “estimation usin”.
- Quality: τ_reg = 0.002 approximately halves the largest splat diameter on average while keeping resampling artifacts visually imperceptible.The coefficient balances rendering and conditioning losses during dictionary construction.
- Quality: Adaptive resampling significantly improves rapidly varying deformations by preventing splats from becoming overstretched.The method targets cases with nonsmooth weighting functions and large Lipschitz constants, where visible artifacts otherwise occur.
- Performance: Adaptive resampling takes 10 s to 1 min 30 s on the evaluated models, while uniformly applying the dictionary takes about 1 s for 160K splats on CPU.Resampling is optional for real-time preview and can be reused across deformations because it operates on the rest shape.
5 Limitations and future work
The method has memory and interpolation limitations, and its current derivations are restricted to linear blend skinning. Future work could trade memory for computation, improve interpolation, and extend resampling across deformation spaces.
- Memory and model scope: Explicitly storing skinning-weight gradients creates memory overhead that increases with the maximum number of rig nodes influencing each splat.Gradients could instead be evaluated from symbolic weight representations on the fly, trading memory usage for computation.
- Memory and model scope: The presented results and equations cover linear blend skinning, while extension to dual quaternion skinning is left as future work.The proposed extension is intended to enable more complex deformations.
- Interpolation: The lookup table uses nearest-entry selection, while improved interpolation of splat attributes remains an open direction.The paper specifically identifies opacity consistency and rotation interpolation as unresolved issues, with Wasserstein barycenters considered too costly in practice.
- Resampling: Resampling is performed in rest space and can be reused for neighboring deformations, but broader deformation-space coverage and faster GPU execution remain future directions.The authors also propose tuning a per-splat elastic factor to reduce elongation selectively for artifact-causing splats.
- Conclusion: The method interactively deforms Gaussian splats without a mesh proxy, specific re-optimization, or view-specific projections, with rest-space resampling improving deformed-shape quality.
Splat count
The experiments examine deformation quality, gradient recovery, and adaptive resampling across large-scale and rapidly varying deformation fields. The method removes holes caused by rigid-only deformation, while resampling addresses overstretched splats that remain under strong variations.
- Gradient recovery: Gradient recovery with 16-neighbor Moving Least Squares depends on selection smoothness: lower smoothness produces gradients too steep for the Gaussian density to estimate reliably.The comparison is against the paper’s automatic differentiable skinning.
- Deformation quality: Large deformations produce holes under rigid-only deformation, whereas the proposed elastic term fixes them without resampling.The examples use smooth selection operations to compute weights and gradients.
- Comparison: Compared with Gao et al., the method reproduces similarly scaled deformations and handles large-scale deformation without relying on a mesh proxy.The comparison also applies the method to splats produced by Gao et al.’s re-optimization.
- Adaptive resampling: Adaptive resampling removes artifacts when strong deformation variations overstretch splats after the method resolves rigid-deformation holes.Reported resampling times are 82.2 s for the banana, 81.7 s for the teddy bear, 21.8 s for the rabbit, and 9.4 s for the dinosaur.
6 Justification of the affine approximation
A general deformation maps Gaussian mixtures to distributions that are not generally Gaussian mixtures, so the paper uses local affine approximations to preserve the Gaussian-mixture representation.
- Distributional formulation: Under a diffeomorphism, the deformed density is defined through the pushforward measure and the Jacobian of the deformation.
- Distributional formulation: A nonlinear deformation of a Gaussian mixture generally does not remain a Gaussian mixture, including the deformation field induced by nonlinear skinning weights.
- Affine case: When the deformation is affine, the resulting distribution remains a Gaussian mixture with an explicit transformed-Gaussian expression.
- Local approximation: The method therefore approximates the deformation field locally on each Gaussian using the first-order Taylor expansion at its center.Each Gaussian is mapped by the corresponding local affine transformation.
7 Jacobian of the deformation field
The deformation Jacobian is derived from normalized skinning weights by applying the product rule and separating rigid and elastic contributions.
- Jacobian formulation: The Jacobian J_F is the spatial derivative of the deformation field with respect to the splat center p.
- Weight differentiation: The derivation begins by introducing normalized skinning weights and differentiating their weighted deformation expression.
- Rigid component: The first product-rule term corresponds to the rigid component of the deformation field.
- Elastic component: The second product-rule term is computed separately to obtain the remaining contribution to the deformation gradient.
8 Hessian of the 𝑘-th coordinate of the deformation field
The section derives the Hessian of each deformation-field coordinate and uses it to assess when the first-order approximation remains valid.
- The Hessian derivation applies the product rule twice to the deformation-field terms.
- K_j is defined as the gradient of G_j.
- When weights sum to one, K_j reduces to ∇2w_j and G_j equals ∇w_j.
- The first-order approximation is valid when the second-order Taylor term of F remains small.
9 Maximum number of rig node per splat
The experiments limit each Gaussian splat to two influencing rig nodes, but this constraint affects 8.5% of splats in a RigAnything scenario.
- 8.5% of Gaussian splats are affected by limiting influence to two rig nodes, typically around hands and feet.These splats can detach from one of their nodes; increasing the limit to three rig nodes addresses the issue.
- The two-node limit reduces the number of per-splat attributes required by the experiments.
- The paper suggests adding another rig-node and weight pair or introducing mock rig nodes tracking weighted barycenters.
10 Interpolation in dictionary lookup
The resampling dictionary uses interpolation after deformation, but at its tested resolution linear interpolation does not significantly improve over nearest-neighbor lookup.
- Nearest-neighbor interpolation is preferred because linear interpolation does not significantly improve results at the dictionary’s tested resolution.The dictionary is precomputed and used to selectively resample deformed Gaussians.
- The technique is evaluated with skinning weights from RigAnything, where 8.5% of Gaussian splats are significantly influenced by more than two rig nodes.
11 Range of eigenvalues
The dictionary discretization is guided by the range of Gaussian eigenvalue ratios, while gradient estimation is compared against a KNN-based reconstruction across feather values.
- Range of eigenvalues: The eigenvalue ratio λmax/λmin is typically between 1 and 10^4 but reaches up to 10^12 in worst cases.This range determines the coverage of the precomputed resampling dictionary.
- Extra deformation results: Additional examples reproduce deformations comparable in scale and nature to those reported by Gao et al.
- Gradient estimation: Moving Least Squares with nearby splats can estimate skinning-weight gradients when weights are known at sampled positions.The implementation uses a KD-tree to select the 16 nearest neighbors.
- Gradient estimation: For larger feather values, KNN gradient reconstruction can be used, but care is needed when splat density is low relative to gradient variation.The experiment treats differentiable skinning gradients as the comparison ground truth.
14 Comparison with Gao et al.
The comparison materials show that the method avoids deformation holes without requiring a mesh proxy and remains compatible with Gaussian splat inputs from different optimizers. Additional analyses characterize resampling fidelity, lookup-table interpolation, dictionary regularization, and resampling-threshold trade-offs.
- Comparison with Gao et al.: The method applies to Gaussian splat inputs from different optimizers, unlike the comparison with Gao et al.The authors identify this adaptability as a notable advantage and separately note that their method does not need a mesh proxy.
- Additional resampling results: Resampling preserves appearance with low, spatially distributed error, although rendered views are not perfectly preserved.
- Additional resampling results: Linear interpolation between four lookup-table entries does not appear to improve results over selecting the closest entry.
- Additional resampling results: The choice of τreg and ε exposes accuracy–efficiency trade-offs: larger τreg lowers deformation error, while smaller ε removes more artifacts but increases resampling cost.The supplementary analysis justifies τreg = 0.002 as a compromise; changing ε from 0.1 to 0.01 causes a 1.5x to 3x slowdown.
- Comparison with Gao et al.: Naïve rigid deformation creates holes, whereas the proposed method removes these artifacts and produces the intended deformation.