ChromaGS: Text-Driven Semantic Editing of 4D Gaussian Avatars

I. INTRODUCTION

• Diffusion-based avatar editing can introduce unintended modifications, struggle with precise localization, and lack deterministic outputs.Users may want to change attributes such as eye color, skin tone, or hair color without altering identity.Source: §I · p. 1
• ChromaGS provides real-time, language-guided semantic color editing of 3D Gaussian head avatars, with modifications specified in natural language and shown instantly.Source: §I · p. 1
• Its approach combines learned soft semantic region assignments, base-color and residual decomposition, and a two-stage language pipeline supporting absolute and relative color changes.The reported experiments demonstrate deterministic editing with faithful appearance preservation across diverse subjects.Source: §I · p. 1

II. RELATED WORK

• Diffusion-based text editing methods for head avatars require per-edit optimization, may introduce unintended changes, and cannot guarantee deterministic outputs.Other cited approaches use diffusion priors or open-vocabulary grounding for general Gaussian scenes, or score distillation to generate stylized heads.Source: §II · p. 1
• Palette-based recoloring supports interactive edits but the cited methods do not model animation, expression-dependent appearance, or semantic face regions.Semantic Gaussian methods cited here target general static scenes rather than animatable avatars or language-driven color specification.Source: §II · p. 2
• ChromaGS combines learned semantic assignments and color decomposition for region-specific editing consistent across expressions and viewpoints.The cited related work also identifies SAM3 as the source of semantic masks for the editing pipeline.Source: §II · p. 2, Figure 4 · p. 2

A. Semantic Masks Extraction

• SAM3 generates text-prompted binary masks for each semantic class and training frame, with smaller regions prioritized to make masks mutually exclusive.Iris masks are derived from eye masks using ellipse detection and a warped annular template; supervision aggregates across hundreds of frames, absorbing occasional segmentation noise.Source: A. Semantic Masks Extraction · p. 2

B. Text-Guided Color Editing

• The two-stage architecture decouples semantic parsing from color resolution, assigning simpler, well-defined tasks to each component to improve robustness.Source: B. Text-Guided Color Editing · p. 2
• Flan-T5-Small parses instructions into structured region, color, and edit-type strings, a simpler output space to learn than direct RGB prediction.Multiple edits are separated by semicolons.Source: B. Text-Guided Color Editing · p. 2
• For absolute instructions, a sentence encoder matches color descriptions by cosine similarity to precomputed, region-specific palettes.The palettes encode anatomically plausible colors, including Fitzpatrick-scale skin tones and natural iris colors based on melanin distributions.Source: B. Text-Guided Color Editing · p. 2
• For relative instructions, the resolver converts the current region color to HSV and applies deterministic modifiers, such as lowering V for “darker.”It increases S for “more saturated.”Source: B. Text-Guided Color Editing · p. 2

C. Semantic-Aware Gaussian Splatting

• The avatar representation binds Gaussians to FLAME for expression-driven animation with linear blend skinning.Each Gaussian is characterized by position, scaling, rotation, opacity, and spherical-harmonic color coefficients.Source: C. Semantic-Aware Gaussian Splatting · p. 2
• Gaussian primitives receive soft semantic class assignments, supervised by differentiable rasterization and pixel-wise cross-entropy against blurred masks.Blurring and renormalizing the binary masks softens supervision at imprecise region boundaries.Source: C. Semantic-Aware Gaussian Splatting · p. 3
• Class-level base colors average masked pixels, while per-Gaussian residuals retain texture, shading, and specular detail and vary with facial expressions.The residuals use learned spherical-harmonic coefficients combined with expression-dependent blendshapes.Source: C. Semantic-Aware Gaussian Splatting · p. 3
• Rendered color combines semantic probabilities, class base colors, and residuals; editing replaces a target region’s base color while unedited regions and local details remain.Soft semantic assignments smooth transitions at region boundaries.Source: C. Semantic-Aware Gaussian Splatting · p. 3
• Cross-Region Adaptation weights mask channels so related regions can shift together; the skin class also uses lips (0.3) and eyebrows (0.2) to avoid pigment disconnects.The skin-tone example describes coherent shifts in pigment-correlated regions.Source: C. Semantic-Aware Gaussian Splatting · p. 3
• Across skin, hair, and eye edits for three subjects, ChromaGS preserves identity and fine detail, while GaussianAvatar-Editor [5] and InstructPix2Pix [22] show color bleeding and oversaturation.The cited comparison notes preserved specular highlights and skin texture.Source: Figure 5 · p. 3

D. Training Objective

• Training combines standard L1 photometric loss with cross-entropy semantic loss, using λrgb = 1.0 and λsem = 0.1 in all experiments.Source: D. Training Objective · p. 3

IV. EXPERIMENTS

• Evaluation covers FaceScape and INSTA scenes, with 14 color edits per scene and identity, CLIP, and context-preservation metrics.The setup yields approximately 10k edited frames per method at 512×512 resolution.Source: §IV · p. 3
• ChromaGS applies subsequent edits in ∼50 ms after one-time ∼800 s training, whereas GaussianAvatar-Editor requires ∼800 s optimization for every edit.InstructPix2Pix processes frames independently via diffusion without per-subject optimization.Source: Table I · p. 4
• ChromaGS achieves significantly better identity and context preservation, while the baselines score higher on CLIP.CLIP measures global image-text alignment and may reward visually dominant color shifts even when they spill into unintended regions.Source: Table II · p. 4

A. Comparison with Baselines

• Across subjects and regions, ChromaGS produces consistent localized changes while preserving identity and fine details across expressions and viewpoints.The 3D Gaussian representation propagates edits across expressions and viewpoints.Source: A. Comparison with Baselines · p. 4
• ChromaGS achieves substantially higher identity and context preservation than the baselines, which score slightly higher on CLIP despite less accurate edits.The authors attribute the CLIP difference to its holistic image-text scoring, which can reward global color shifts that spill beyond the target.Source: A. Comparison with Baselines · p. 4
• Without semantic supervision, edits spill into unrelated areas; the full model produces clean, confined changes.Examples include eye color tinting hair and tongue, and lipstick affecting skin.Source: Figure 6 · p. 4
• After one-time ∼800 s training, ChromaGS applies each edit in ∼50 ms, while GaussianAvatar-Editor repeats ∼800 s optimization for every edit.InstructPix2Pix edits frame by frame and introduces severe temporal flickering.Source: A. Comparison with Baselines · p. 4

B. Ablation Studies

• Omitting semantic supervision prevents Gaussians from learning distinct semantic region assignments.The ablation sets λsem = 0.Source: B. Ablation Studies · p. 4
• Without semantic supervision, edits erroneously affect unrelated facial regions.The passage attributes this leakage to the lack of distinct region assignments.Source: B. Ablation Studies · p. 4
• Lsem regularizes primitives into coherent semantic clusters, anchoring color modulation to intended targets.The authors describe this as robust semantic disentanglement.Source: B. Ablation Studies · p. 4

V. LIMITATIONS AND FUTURE WORK

• Localization quality depends on semantic segmentation, while the method supports uniform color shifts rather than texture, material, or geometric edits.High-frequency structures such as hair may also lack photorealism because residual-based decomposition struggles with intricate details.Source: §V · p. 4
• Without Lsem, color changes spread beyond the target region and degrade all three metrics, including CLIP.The ablation attributes the CLIP drop to leakage diluting the edit rather than amplifying it.Source: Table III · p. 5
• Out-of-palette tones default to the nearest palette entry, and reconstruction quality is bounded by FLAME tracking and the base Gaussian avatar.Source: §V · p. 5
• Future work includes spatially varying material edits, richer appearance priors, open-set region discovery, and palette-free color strategies.Source: §V · p. 5

• ChromaGS enables real-time, language-driven color editing of animatable 3D Gaussian avatars.Source: §VI · p. 5
• Coupling learned segmentation with color decomposition enables localized, deterministic control without per-edit optimization.Source: §VI · p. 5
• A two-stage pipeline translates natural language into precise absolute and relative color adjustments.Source: §VI · p. 5
• A single forward pass supports high-fidelity avatar customization for interactive gaming and virtual try-on.Source: §VI · p. 5