PaperlayerView exact revision on arXiv

Source-linked AI summary

ManifoldSplat: Language-Guided Semantic Shape Editing of 3D Gaussian Head Avatars

Antonio Canela, Jordi Sànchez-Riera

arXiv:2610.03599v1cs.CV

TL;DR

Localized geometric editing of monocular head avatars remains difficult to achieve while preserving identity and animation. ManifoldSplat edits within FLAME using a gated feed-forward shape model and targeted Gaussian refinement; evaluations report better prompt alignment, identity preservation, visual cleanliness, and edit locality than the cited baselines.

  • Problem

    Existing text-driven methods lack localized geometric editing of personal monocular avatars that preserves identity and animation rigging.

    Source: §1 · p. 2

  • Method

    ManifoldSplat combines feed-forward, region-gated FLAME shape deltas from DeltaRegion with targeted Gaussian refinement of the edited avatar.

    Source: §1 · p. 2, §3.3 · p. 6, §3.4 · p. 8

  • Results

    ManifoldSplat outperforms GaussianAvatar-Editor and InstructPix2Pix on prompt alignment, identity preservation, and visual cleanliness while being one to two orders of magnitude faster, and improves shape-edit locality over DESCRIBE3D and Semantify.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pipeline reduces each edit to brief refinement of a pretrained avatar while preserving animation fidelity and localized identity-preserving edits.

    Source: §5 · p. 13

    The edit space is bounded by predefined deformation prototypes and FLAME surface geometry, excluding novel structural changes, hair, and appearance attributes such as skin or eye color.

    Source: §4.4 · p. 13

Abstract

from arXiv · show

High-fidelity 3D head avatars have reached near-photorealistic quality. While recent methods enable text-driven manipulation, they struggle to provide fine-grained localized control, often entangling features or lacking geometric consistency. Modifying geometry through natural language currently requires slow per-prompt optimization or compromises identity and rigging. We present ManifoldSplat, the first end-toend framework for language-guided semantic shape editing of animatable 3D Gaussian Splatting avatars reconstructed from monocular videos. By performing edits within the structured FLAME manifold rather than directly optimizing an unstructured Gaussian cloud, we strictly preserve identity and animation. We introduce DeltaRegion, a per-region disentangled Conditional Variational Autoencoder (CVAE) delivering feedforward shape deltas, alongside a refining stage to recover view-consistent details. ManifoldSplat reconstructs and edits an avatar in ~90 seconds on a consumer GPU, rendering at ~800 FPS. Extensive evaluations demonstrate our approach sets a new state-of-the-art in localized prompt alignment, geometric coherence, and identity preservation. Project page and code: https://a-canela.github.io/manifoldsplat/

1 Introduction

Text-driven editing of monocular head avatars lacks localized geometric control that preserves identity and animation. ManifoldSplat edits within FLAME’s structured identity manifold, using DeltaRegion and targeted Gaussian refinement to produce localized, identity-preserving edits.

1 Introduction

  • Existing methods do not provide localized geometric editing of a personal monocular avatar while preserving its animation rigging and sub-minute latency.Text-to-3D-face methods can fail to preserve personal identity, while distillation-based editors require lengthy per-prompt optimization and can break multi-view consistency.Source: §1 · p. 2
  • FLAME’s identity subspace provides plausible head geometry and downstream animation compatibility, but lacks explicit correspondence to individual facial attributes.The approach therefore learns text-conditioned edits within FLAME while addressing localization and avoiding entanglement with unrelated attributes.Source: §1 · p. 2
  • ManifoldSplat uses a two-stage pipeline: a text-guided FLAME shape edit produces a localized identity delta, then targeted rendering adapts splats to the edited geometry.The refinement addresses characteristic geometric artifacts without full per-edit optimization.Source: §1 · p. 2
  • ManifoldSplat outperforms GaussianAvatar-Editor and InstructPix2Pix on prompt alignment, identity preservation, and visual cleanliness while being one to two orders of magnitude faster.At the shape level, it significantly outperforms DESCRIBE3D and Semantify in edit locality.Source: §1 · p. 2
  • DeltaRegion produces a localized FLAME shape delta from a free-form prompt and source identity in one feed-forward pass.The paper also contributes a synthetic dataset of per-region vertex-displacement edits.Source: §1 · p. 2

2 Related Work

ManifoldSplat combines parametric face models, mesh-anchored Gaussian avatars, and text-driven editing, addressing the gap between general text-to-3D generation and localized editing of a monocularly reconstructed identity.

2.1 Parametric Face Models.

  • Parametric face models provide a shape space, but their globally reduced parameters are entangled and lack explicit correspondence to localized facial attributes.Localized control on the FLAME manifold remains an open challenge addressed by DeltaRegion.Source: §2.1 · p. 3

2.2 Head-Avatar Reconstruction.

  • Mesh-anchored Gaussian avatars support real-time, high-fidelity animation; ManifoldSplat builds on a Gaussian-blendshape formulation whose identity-driven parameterization makes shape edits well defined.Its rendering-side enhancements are described as orthogonal and potentially integrable into other mesh-anchored Gaussian avatars.Source: §2.2 · p. 3

2.3 Text-Driven 3D Face Generation.

  • Text-to-3D face methods generate faces or heads rather than editing an identity reconstructed from a user video, so direct comparison would conflate different tasks.DESCRIBE3D and Semantify serve as shape-level counterparts, though both synthesize from a neutral mesh without source-identity conditioning.Source: §2.3 · p. 3

2.4 Editing 3D Avatars and Scenes.

  • Among published methods, GaussianAvatar-Editor and an InstructPix2Pix-based editor meet the stated constraints of animatable, monocularly reconstructed Gaussian heads and free-form text instructions.Many multi-view editors require calibration or assume a static scene, unlike monocular talking-head video.Source: §2.4 · p. 4

2 Related Work.

  • ManifoldSplat’s overview separates a monocular-video-to-avatar pipeline from DeltaRegion’s gated text-conditioned shape-editing CVAE.The region gate zeroes non-selected latent blocks before decoding.Source: Figure 2 · p. 4

3 Method

ManifoldSplat edits a mesh-bound Gaussian avatar by changing its FLAME identity shape and then refining splats against the edited geometry. Its method combines synthetic per-region supervision, gated prompt-conditioned shape deltas, and targeted rendering adjustments.

3 Method

  • The pipeline takes a short monocular RGB video and free-form prompt, edits the FLAME shape through DeltaRegion, then refines Gaussian parameters for the new geometry.It adopts RGBAvatar as its reconstruction backbone and uses FLAME identity, expression, and pose parameters as separate subspaces.Source: §3 · pp. 4–5

3.1 Background - Preprocessing and Mesh-bound Backbone

  • Mesh-bound Gaussians are anchored to FLAME triangles, so identity-shape deformations propagate to the splats without full re-optimization.The preprocessing stack uses MICA for identity initialization, per-frame FLAME tracking, and foreground extraction.Source: §3.1 · p. 5

3.2 Synthetic Dataset

  • About 130 fine-grained prototypes across six face regions are projected onto source identities, yielding about 500k supervised prompt-edit triplets without per-subject mesh annotation.Each prototype specifies selected FLAME vertices and displacement vectors; prompt variations and edit scales expand the tuples.Source: §3.2 · pp. 5–6

3.3 Stage 1 - Shape Editing

  • DeltaRegion gates six region-specific latent blocks before decoding a FLAME delta; inference is deterministic, with prompt qualifiers and prior scale controlling edit strength.Its conditional VAE fuses frozen text encoding and source-shape embedding, while masked reconstruction and gate supervision train localized edits.Source: §3.3 · pp. 6–7

3.4 Stage 2 - Gaussian Refining

  • Surface consistency keeps splats on the mesh; tanh color activation smooths fields; topology-based warping supplies pseudo-targets, then color-frozen anchored refinement realigns splats.The warp uses shared mesh topology and triangle-barycentric reprojection, while the anchor term limits drift from the warm start.Source: §3.4 · pp. 7–8

4 Experiments

ManifoldSplat is evaluated on full-avatar editing, shape editing, and rendering-enhancement ablations. It tops the reported full-avatar metrics, while qualitative results show localized edits and the limitations section bounds the method’s scope.

4.1 Experimental Setup

  • Evaluation uses 18 natural-language edits per subject across six regions on held-out front-camera frames, and strongly out-of-distribution phrasings are outside the study’s focus.The reported avatar evaluations use 100–300 test frames per subject-edit pair.Source: §4.1 · p. 9, §4 · p. 9

4.2 Quantitative results

  • ManifoldSplat tops every full-avatar metric, with CLIP-D gaps of +5% over GaussianAvatar-Editor and +13% over InstructPix2Pix.Table 1 reports the full-avatar editing comparison.Source: Table 1 · p. 9, §4.2 · p. 9
  • ArcFace is 0.93 vs. 0.76 and 0.52, while TV2D is lowest at 0.0122.The authors attribute the ArcFace result to FLAME-manifold constraints and the TV2D result to avoiding per-frame 2D denoising.Source: §4.2 · p. 10
  • Reconstruction takes ∼60 s once and subsequent edits ∼30 s each, versus ∼800 s per prompt for GaussianAvatar-Editor.InstructPix2Pix processes frames independently and produces no animatable avatar.Source: Table 3 · p. 10, §4.2 · p. 10

4.3 Qualitative results

  • In the full-avatar comparison, InstructPix2Pix shows 2D artifacts and identity drift, GaussianAvatar-Editor barely realizes geometric changes, and ManifoldSplat makes localized edits.ManifoldSplat also preserves identity and surrounding appearance.Source: §4.3 · p. 11, Figure 3 · p. 11
  • For two ear edits, DeltaRegion concentrates displacement on the named region, unlike DESCRIBE3D and Semantify, whose changes spread across the face, neck, or bust.The comparison is qualitative because the baselines generate identities from scratch, so per-vertex distance would conflate locality with identity drift.Source: §4.3 · pp. 11–12, Figure 4 · p. 11
  • Across three subjects, nose prompts range from global elongation to tip and base tweaks, with changes confined to the named region; removing rendering enhancements yields distinct lip artifacts.Figure 6 shows floaters, stray colored splats, and ghosting or color seams when individual enhancements are removed.Source: §4.3 · pp. 12–13, Figure 5 · p. 12, Figure 6 · p. 12

4.4 Limitations

  • Residual FLAME–Gaussian misalignment can amplify rendering artifacts under extreme shape deformations, and the finite prototype library cannot generalize to completely novel structural changes.Edits are limited to FLAME surface geometry; hair and appearance attributes such as skin or eye color are not addressed.Source: §4.4 · p. 13

5 Conclusions

ManifoldSplat edits animatable Gaussian head avatars by separating FLAME-based geometric manipulation from splat rendering. The appendix details its architecture, synthetic data generation, and optimization hyperparameters.

5 Conclusions

  • ManifoldSplat is an end-to-end pipeline for language-guided semantic shape editing of animatable 3D Gaussian head avatars reconstructed from monocular video.It decouples geometric manipulation via DeltaRegion from splat-based rendering.Source: §5 · p. 13
  • Targeted rendering regularizations keep avatars visually coherent under structural deformations and replace costly full re-optimization with brief refining of the pre-trained avatar.The conclusion frames this as a step toward conversational, real-time avatar sculpting.Source: §5 · p. 13

A Implementation Details

  • The implementation appendix covers DeltaRegion architecture, synthetic paired-dataset generation, and hyperparameters for 3D Gaussian Splatting optimization.Source: A Implementation Details · p. 14

A.1 DeltaRegion architecture

DeltaRegion combines a prompt and source FLAME identity to predict an edited shape delta, using text-conditioned, region-gated latent variables for localized edits.

A.1 DeltaRegion architecture

  • A frozen Flan-T5 text encoder and an MLP source-shape embedding are fused by cross-attention to form shape-conditioned semantic context.The text encoding supplies token features, while the source shape is represented as a 512-D token.Source: §A.1 · p. 14
  • A text-only gate assigns softmax weights to six regional latent blocks, which are decoded with the source-shape embedding into a 300-D shape delta.The training-only posterior uses the ground-truth delta; the prior is conditioned on the fused text and source-shape representation.Source: §A.1 · p. 14
  • The edited shape is formed by adding the predicted delta to the source identity, and inference uses the prior mean with a scale factor of 1.2.Varying the scale continuously modulates edit strength, while the mapping remains deterministic per prompt.Source: §A.1 · pp. 14–15
  • DeltaRegion is trained for 100 epochs with AdamW at a learning rate of 10^-4 and batch size 256 on one NVIDIA RTX 5090.Source: §A.1 · p. 15

A.2 Training data

The training set pairs source identities, prompts, and target FLAME shapes using region-localized synthetic edits and prompt variations.

A.2 Training data

  • About 500,000 supervised triplets are generated from 300 MICA-fitted LYHM identities, edit templates, three edit scales, and five paraphrased prompts per tuple.15% of the triplets is held out for validation.Source: §A.2 · p. 15
  • About 130 fine-grained edit templates cover six FLAME regions, with displacement vectors applied to canonical vertex sets and an outside-region penalty to limit leakage.The resulting target deltas are recovered by iterative projection onto the FLAME shape basis.Source: §A.2 · p. 15
  • Generating the data takes about 70 hours, and DeltaRegion training takes about 40 hours, each on one RTX 5090.Source: §A.2 · p. 15

A.3 Full training objective

The appendix specifies DeltaRegion’s training objective, the warped-render refinement procedure, the Gaussian-avatar configuration, and the evaluation metrics.

A.3 Full training objective

  • The full objective combines vertex-space supervision, parameter supervision, an ELBO term, and gate cross-entropy, with vertex losses targeting the edited region and anchoring its complement to the source.The KL warm-up and per-dimension free-bits floor are specified as part of the objective.Source: §A.3 · p. 15

A.4 Warped refining: per-pixel resampling map

  • Warped refining maps each edited-mesh pixel to the unedited render using shared triangle and barycentric coordinates, producing view-consistent pseudo-ground-truth images.The topology-derived warp is texture-insensitive and avoids optical-flow bracketing failures in low-texture regions.Source: §A.4 · p. 16
  • An anchor loss keeps non-frozen Gaussian parameters near their warm start, while spherical-harmonic DC color channels remain frozen during refining.The refining pass reuses the subject’s base-reconstruction checkpoint and RGBAvatar configuration.Source: §A.4 · p. 16, §A.5 · p. 16

A.5 3DGS backbone and refining hyper-parameters

  • The base and refine passes use the RGBAvatar backbone with 20 identity blendshapes layered over 129 FLAME bases; refining starts from the shared per-subject reconstruction checkpoint.Source: §A.5 · p. 16

A.6 Evaluation metrics

  • CLIP-D measures alignment between visual and textual changes, while ArcFace cosine similarity is the strict identity-preservation metric and isolates identity drift from texture and pose differences.The metrics are evaluated over frames; higher CLIP-D and ArcFace are better.Source: §A.6 · p. 17
  • TV2D averages horizontal and vertical neighboring-pixel differences across RGB frames and serves as a no-reference proxy for high-frequency rendering artifacts.Lower TV2D is better, and normalization makes scores comparable across resolutions.Source: §A.6 · p. 17

B.1 Per-region quantitative breakdown

The per-region breakdown finds rendering quality essentially unchanged across edited regions, while identity and CLIP-D scores vary with the spatial footprint of each edit.

B.1 Per-region quantitative breakdown

  • σ=8×10−5 for TV2D, indicating essentially flat rendering quality across regions.The authors attribute this to warped refining and the surface-consistency regularizer.Source: §B.1 · pp. 17–18, Table 4 · p. 18
  • ArcFace remains above 0.88 even for the most severe edit, versus 0.76 for GaussianAvatar-Editor on full edits.Peripheral edits score highest because they leave the identity-bearing inner face untouched; reshaping core features produces a larger identity shift.Source: §B.1 · p. 18, Table 4 · p. 18
  • CLIP-D reflects an edit’s silhouette footprint rather than its semantic fidelity, with peripheral edits projecting further onto the textual direction than localized lip prompts.The passage describes this as a known property of CLIP-D.Source: §B.1 · p. 18, Table 4 · p. 18

B.2 Edit-locality heatmaps

Per-vertex heatmaps show that representative edits concentrate on their named FLAME regions, with smooth semantic deformations rather than rigid mask-shaped changes.

B.2 Edit-locality heatmaps

  • Heatmap color encodes each vertex’s normalized per-mesh displacement magnitude, from blue to red, for one prompt in each of six FLAME regions.The examples use the source identity of subject 104_EXP_5 from NeRSemble [16].Source: Figure 7 · p. 18, §B.2 · p. 18
  • The six edits concentrate on their named regions, from close-set eyes and big lips to a double chin, nose bridge, forehead, and earlobe.The reported energy remains almost exclusively on the target region, leaving adjacent structures undisturbed.Source: §B.2 · pp. 18–19
  • The nose-bridge shift tapers smoothly along the nasal ridge rather than clipping to the canonical nose mask, supporting sequential, composable edits without interference.The authors attribute this semantic locality to the learned FLAME-manifold correlation structure rather than mechanical reproduction of the authored template.Source: §B.2 · p. 19

B.3 Extended qualitative results

Extended examples show fine-grained edits across diverse subjects and attributes, including non-frontal, in-the-wild nose edits that retain surrounding structure and view-dependent appearance.

B.3 Extended qualitative results

  • Across six subjects and three prompts per subject, edits vary facial shape while preserving unedited features such as eyebrow shape, skin texture, and eye color.Examples include changing interocular distance without touching the lips or hairline, and varying forehead and jaw shapes on one subject.Source: §B.3 · p. 19, Figure 8 · p. 20
  • Because edits act on the mesh-bound avatar, rendered portraits retain view-dependent appearance and remain drivable by the tracker’s expression and pose stream at >800 fps.The passage notes that downstream animation inherits the underlying perturbed FLAME geometry.Source: §B.3 · p. 19
  • On an in-the-wild monocular video without studio setup or controlled lighting, near-profile views show coherent silhouettes and view-dependent shading alongside localized nose edits.The profile views expose aquiline, protruded-tip, and recessed-bridge changes that are largely unnoticeable frontally.Source: §B.3 · p. 19, Figure 9 · p. 21
Loading 2610.03599v1…