Source-linked AI summary
Reparametrizing 3D Gaussian Splatting for Real-Time Palette-based Color and Luminance Editing
Cheng-Kang Ted Chao, Yotam Gingold
TL;DR
Palette-based editing in 3DGS lacks sparse, localized pixel-level control and independent luminance editing because primitive-space decompositions become dense after alpha blending. The paper reparameterizes pretrained 3DGS into a view-space palette-weight representation and uses a real-time constraint solver for recoloring, tone curves, and pixel constraints. The resulting edits are sparser and more localized than prior palette-based 3DGS methods while remaining consistent across views and compatible with standard viewers.
Problem
Prior primitive-space palette decompositions lose pixel-level sparsity after alpha blending, while existing methods lack color-aware tone curves and efficient pixel-level constraints.
Method
The method finetunes pretrained 3DGS color SH into view-dependent palette-weight SH, supervises sparse view-space decompositions, and solves palette, tone-curve, and pixel constraints with IRLS and dBCD.
Results
The framework improves edit quality and interactivity over prior palette-based 3DGS methods while providing independent luminance control, view-consistent pixel constraints, and standard-viewer compatibility.
Takeaways & Limitations
View-space palette decomposition enables sparse, localized 3DGS edits with direct palette, per-palette tone-curve, and pixel-level controls across novel views.
Takeaways & Limitations
The representation requires brief finetuning from pretrained vanilla 3DGS, tone-curve edits require brief finetuning for bake-back, and semantic or object-level constraints are unsupported.
Abstract
from arXiv · showhide
Professional color editing requires precise control over both color (hue and saturation) and lightness, ideally through separate, independent controls. We present a real-time interactive color editing framework for 3D Gaussian Splatting that supports palette-based recoloring, per-palette tone curves for color-aware luminance adjustment, and pixel-level color constraints. Rather than training a new representation from scratch, we reparameterize the spherical harmonics of a pretrained vanilla 3DGS to encode view-dependent palette weights. We simultaneously solve for weights and palette colors via a loss based on image-space sparsity. Luminance editing is realized as a per-pixel weight shift along the achromatic axis, which we show is equivalent to a per-pixel palette-aware luminance edit. This view-space formulation addresses a core limitation of prior primitive-space methods, where alpha-blending breaks per-Gaussian sparsity and causes edits to bleed into unintended regions. Our edits run in tens of milliseconds via an iteratively reweighted least squares and damped block-coordinate descent that couples tone curves and palette shifts under view-space sparsity. Our representation can be efficiently baked back into a vanilla 3DGS, preserving compatibility with standard viewers. We demonstrate sparser, more localized edits than prior palette-based 3DGS methods, while enabling independent luminance control per palette color and view-consistent pixel-level constraints, capabilities previously unavailable for 3DGS.
1 Introduction
The paper extends palette-based color editing to 3DGS with view-consistent, independent color and luminance controls. It reparameterizes pretrained 3DGS into a sparse view-space palette representation and supports real-time editing with standard-viewer compatibility.
- Professional 3D color grading requires selective color-range edits with independent lightness control that remain consistent across viewpoints.
- Primitive-space palette decompositions lose pixel-level sparsity during alpha blending, causing dense weights and edits to bleed into unintended regions.
- The method reparameterizes pretrained 3DGS color SH into view-dependent palette-weight SH while preserving the original scene geometry.
- The framework provides direct palette manipulation, per-palette tone curves, and pixel-level color constraints with edits that propagate across novel views.
- The framework improves edit quality and interactivity over prior palette-based 3DGS methods while preserving compatibility with standard viewers.Palette edits bake back at essentially no cost, while luminance edits require a brief finetuning stage of a few seconds.
2 Related Work
Prior neural scene-editing methods support palette manipulation but often incur preprocessing or rendering costs, while primitive-space 3DGS decompositions do not preserve pixel-level sparsity. This work instead performs decomposition after splatting while retaining standard 3DGS viewer compatibility.
- NeRF palette-editing methods support real-world scenes but require expensive preprocessing and slow rendering.
- Recent palette-based 3DGS methods achieve real-time performance by decomposing each Gaussian into mixtures of global palette colors.
- Primitive-space decomposition does not survive alpha blending, so per-Gaussian sparsity becomes non-sparse at pixels and can cause unintended edits.
- The proposed approach reparameterizes pretrained 3DGS so decomposition occurs at pixel level after splatting, while remaining compatible with standard viewers through bake-back.
3 Background and Motivation
Primitive-space palette decomposition becomes dense after alpha blending, undermining localized editing and creating additional problems for pixel constraints. These limitations motivate view-space palette weights and regularization.
- Primitive-space methods represent each Gaussian’s diffuse color as a mixture of global palette colors with sparse per-Gaussian weights.
- Alpha blending many Gaussians at one pixel breaks sparsity, producing nonzero weights across most palette colors and bleeding edits into unintended regions.
- In sparse-coverage regions, the residual contribution becomes black and dilutes editing control.
- Pixel-level color constraints are ill-posed because the palette affects only diffuse color while specular highlights remain independent.
- Palette-only optimization can take over 10 seconds on GPU and still bleed, while jointly optimizing palette and specular SH converges faster but causes severe bleeding.
4 Method: Representation and Finetuning
The method reparameterizes pretrained 3DGS spherical harmonics as view-dependent palette weights, supervises them with sparse barycentric decompositions, and compresses higher-order weights through rank-1 factorization.
- 4.1 View-space Palette Decomposition: View-space palette weights are splatted per pixel and converted to colors as P^T W(x;θ,ϕ), while frozen geometry preserves compatibility with vanilla 3DGS.Only per-Gaussian SH coefficients are finetuned; positions, covariances, and opacities remain fixed, and palette edits can be baked back into vanilla 3DGS.
- 4.2 Barycentric Supervision: Barycentric supervision decomposes each frozen rendered color over tetrahedral wedges that fan around the grey axis, producing sparse, color-consistent pixel weights.The palette contains fixed black and white anchors plus learnable chromatic vertices, whose hue ordering is maintained through HSL parameterization and normalized hue gaps.
- 4.2 Barycentric Supervision: Softplus simplex projection and low-temperature softmax select the wedge whose reconstructed color best matches the query, while target weights retain raw barycentric coordinates.The final targets mix raw barycentric coordinates across wedges using reconstruction-error weights with τ = 10^-9.
- 4.3 Joint Palette and Weight SH Finetuning: Joint finetuning combines barycentric color supervision with positivity, compactness, and hue-separation regularizers to avoid negative weights and palette collapse.Compactness pressure is scheduled late so the palette can first expand toward the scene’s color distribution before tightening.
- 4.4 Rank-1 Factorization of Higher-Order Weight SH: Rank-1 factorization compresses higher-order weight SH with negligible reconstruction loss and visually comparable recoloring, despite slight non-identical details.For K = 6 and L = 3, the factorization uses 25 scalars per Gaussian, roughly half of vanilla 3DGS’s 48.
5 Method: Editing
The framework edits pretrained 3DGS through palette, tone-curve, and pixel constraints in view space, jointly optimizing sparse palette changes and luminance adjustments for real-time interaction.
- Three constraint types—pixel-level colors, palette vertices, and per-palette tone curves—feed one optimization while splatted weights and Gaussian parameters remain fixed.Users can click rendered pixels, modify palette colors, or place tone-curve control points.
- Palette colors P and per-pixel splatted weights W define edited colors, while luminance editing redistributes weight mass between black and white palette entries.The luminance shift is computed from the original per-pixel BT.709 luminance and tone-curve samples; chromatic palette colors dominate the shift through a small black-white scaling factor.
- The weight shift alone is equivalent to a per-pixel luminance edit proportional to each palette color’s chromatic weight, preserving the chromatic component exactly.
- The constrained optimization minimizes palette and tone-curve changes subject to image, palette, and curve constraints, with an L2,1 term inducing global sparsity across palette colors and curves.Tone curves are pinned at 0 and 1, and the image loss aggregates squared RGB residuals at constrained pixels.
- IRLS handles the non-differentiable sparsity term, while damped block-coordinate descent alternates tone-curve and palette updates for their coupled RGB residuals.The asymmetric updates project the tone-curve block onto luminance while exposing the palette block to the full residual; damping prevents oscillation, with typical convergence in 5–15 outer iterations.
6 Results and Evaluation
The evaluation demonstrates real-time palette, luminance, and pixel-level editing across scenes, while preserving novel-view consistency and substantially reducing representation storage.
- The method supports palette recoloring, palette-aware luminance editing, and pixel-level color constraints that propagate consistently across novel views.
- The interface sustains approximately 200 FPS on a MacBook Pro with Apple Silicon, using five or six palette colors across examples.
- The evaluation covers reconstruction fidelity, edit locality and consistency, pixel-level editing, bake-back quality, and a Lab-space variant motivating RGB editing.
- Our rank-1 factorization matches full weight spherical harmonics at approximately 60% of vanilla 3DGS storage, while clamping trades approximately 1 dB PSNR for non-negative edit-friendly weights.
- Primitive-space baselines require 3.75–13.02 seconds per pixel-level edit and still exhibit color bleeding.
7 Conclusion
The paper presents a view-space palette decomposition for real-time 3DGS color grading and reports broad editing effectiveness across scenes and scenarios. It also identifies finetuning, tone-curve bake-back, and semantic editing as remaining limitations.
- The framework reparameterizes pretrained 3DGS into view-space palette weights and jointly solves palette, tone-curve, and pixel-level constraints in tens of milliseconds per edit.
- Extensive experiments demonstrate effectiveness across a variety of scenes and editing scenarios.
- The representation requires brief finetuning, tone-curve edits cannot be baked back in closed form, and semantic or object-level editing constraints are unsupported.