PaperlayerView exact revision on arXiv

Source-linked AI summary

CaricatureGS: Exaggerating 3D Gaussian Splatting Faces With Gaussian Curvature

Eldad Matmon, Amit Bracha, Noam Rotstein, Ron Kimmel

arXiv:2601.03319v1cs.GRcs.AIcs.LG

TL;DR

Photorealistic, controllable 3D face caricatures remain challenging, and natural-expression training alone does not faithfully model caricature deformations. CaricatureGS combines curvature-based mesh exaggeration with 3D Gaussian Splatting trained on real and warped pseudo-ground-truth views. It reports improved fidelity and identity retention, continuous intensity control, and localized edits, while noting residual visual and expression-coverage limitations.

  • Problem

    Photorealistic, controllable 3D caricaturization remains an open challenge, and caricature deformations lie outside the distribution of natural expression dynamics.

    Source: §1 · pp. 1–2

  • Method

    CaricatureGS deforms a FLAME mesh with a weighted Poisson equation and trains mesh-rigged Gaussians using alternating real views and locally warped pseudo-ground-truth caricature images.

    Source: §1 · p. 2

  • Results

    The method consistently surpasses its baseline across all evaluation measures and produces identity-preserving, view-consistent exaggerations with continuous intensity control.

    Source: §4.4 · p. 7

  • Takeaways & Limitations

    One controllable 3DGS avatar can represent natural and caricatured appearance, support local feature edits, and retain identity under large exaggerations.

    Source: §1 · p. 2, §7 · p. 8

    Residual specularity and eyelid inaccuracies remain, hair can be over-smoothed, and the deformed FLAME mesh does not fully span facial expressions.

    Source: §6 · p. 8

Abstract

from arXiv · show

A photorealistic and controllable 3D caricaturization framework for faces is introduced. We start with an intrinsic Gaussian curvature-based surface exaggeration technique, which, when coupled with texture, tends to produce over-smoothed renders. To address this, we resort to 3D Gaussian Splatting (3DGS), which has recently been shown to produce realistic free-viewpoint avatars. Given a multiview sequence, we extract a FLAME mesh, solve a curvature-weighted Poisson equation, and obtain its exaggerated form. However, directly deforming the Gaussians yields poor results, necessitating the synthesis of pseudo-ground-truth caricature images by warping each frame to its exaggerated 2D representation using local affine transformations. We then devise a training scheme that alternates real and synthesized supervision, enabling a single Gaussian collection to represent both natural and exaggerated avatars. This scheme improves fidelity, supports local edits, and allows continuous control over the intensity of the caricature. In order to achieve real-time deformations, an efficient interpolation between the original and exaggerated surfaces is introduced. We further analyze and show that it has a bounded deviation from closed-form solutions. In both quantitative and qualitative evaluations, our results outperform prior work, delivering photorealistic, geometry-controlled caricature avatars.

1. Introduction

CaricatureGS combines curvature-driven face deformation with 3D Gaussian Splatting to produce photorealistic, controllable caricatures. It addresses the fidelity gap between natural-expression training and caricature deformations while building on prior neural avatar and mesh-exaggeration approaches.

1. Introduction

  • Directly treating caricature deformation as an expression yields low fidelity, so the method trains one Gaussian set with alternating real and warped pseudo-ground-truth views.Per-triangle local affine transforms warp frames to the caricature projection, and a spatial mask freezes occlusion-affected Gaussians during synthetic-supervision steps.Source: §1 · pp. 1–2, Figure 2 · p. 3
  • The representation supports continuous caricature intensity and localized edits while retaining identity under exaggeration.Its intensity control uses efficient interpolation between the original and exaggerated surfaces; localized edits can exaggerate one feature while leaving unrelated regions unchanged.Source: §1 · p. 2

2.1. Representation for 3D Head Avatars

  • Neural head-avatar methods use implicit fields, NeRFs, and 3D Gaussian representations to support controllable or photorealistic rendering and animation.Examples include morphable-model or deformation-field approaches, dynamic NeRFs, and mesh-bound or blendable Gaussian avatars.Source: §2.1 · p. 2

2.2. Mesh Deformation and Exaggeration

  • Mesh exaggeration spans curvature- and Poisson-based geometry processing, learned deformation from examples, and style-transfer or generative approaches.The cited approaches include identity-preserving 3D caricature generation and 3DGS editors that primarily alter appearance while leaving geometry unchanged.Source: §2.2 · pp. 2–3

3. Method

CaricatureGS combines curvature-driven face deformation with 3D Gaussian Splatting, using warped caricature images and alternating real and synthetic supervision to train one shared Gaussian set. Its features include efficient control over exaggeration intensity and localized edits.

3. Method

  • The pipeline starts from a multiview video and FLAME-fitted mesh, then rigs Gaussians to neutral and caricatured meshes and optimizes them with alternating photometric losses.The shared Gaussian set supports real-time rendering across exaggeration levels and region-specific edits.Source: §3 · pp. 3–4

3.1. Surface Caricaturization

  • Curvature-driven deformation is formulated as a weighted Poisson equation on a temporally consistent FLAME mesh to exaggerate facial geometry while preserving temporal coherence.The discrete solution uses a least-squares formulation and can impose prescribed positions on selected vertices.Source: §3.1 · p. 4

3.2. GT∗Generation via Local Affine Transforms

  • Local Affine Transformations warp each input frame according to corresponding projected triangles, producing masked pseudo-ground-truth caricature images for 3DGS supervision.Masks mark occluded regions and hair boundaries where pixels cannot be warped reliably.Source: §3.2 · p. 4

3.3. CaricatureGS Training

  • A single shared Gaussian set is optimized with alternating real-frame and caricatured GT* supervision, while masks exclude pixels without reliable warped correspondences.This joint training also lets original frames fill occlusion-induced holes and capture hair details absent from direct GT* supervision.Source: §3.3 · p. 5

3.4. CaricatureGS Features

  • Vertex-wise interpolation between the original and precomputed caricature meshes supports intermediate exaggeration levels without repeatedly solving the curvature-weighted Poisson problem.The approximation has zero residual at the endpoints and a small, negligible maximum deviation near the midpoint in practice.Source: §3.4 · p. 5
  • Constrained curvature deformation exaggerates a selected facial region and blends smoothly with the rest, allowing the same rigged Gaussian set to preserve identity and global shape.Figure 4 visualizes localized, semantically controlled facial exaggerations.Source: §3.4 · p. 6, Figure 4 · p. 6

4. Experiments

Experiments evaluate photorealistic caricature rendering and identity preservation on NeRSemble against SurFhead, using image, identity, and view-consistency metrics. Results report stronger scores and visually controlled caricatures, while a diffusion editor exhibits multi-view defects.

4. Experiments

  • Experiments use NeRSemble and the same train/validation/test partition as SurFhead, with 120,000 training iterations.NeRSemble includes 16 synchronized cameras and scripted, emotion-driven, expression-driven, and self-reenactment sequences.Source: §4.1 · p. 6, §4 · p. 6

4.4. Results

  • Our method consistently surpasses the baseline across all reported measures.The comparison covers caricature intent, identity preservation, and view consistency; higher is better.Source: Table 1 · p. 6, §4.4 · p. 7

4.2. Baseline

  • SurFhead is the comparison baseline: it is trained on the original sequence, then its mesh is exaggerated at inference.The authors select it as the most suitable comparison because it achieves state-of-the-art head reconstruction and reenactment and can handle mesh deformations through JBS.Source: §4.2 · p. 6

4.3. Metrics

  • CLIP-I, CLIP-D, and SD assess caricature intent; DINO and CLIP-D assess identity or localization, while CLIP-C assesses novel-view consistency.SD measures diffusion-model noise prediction error, with higher scores indicating greater consistency with its training distribution.Source: §4.3 · pp. 6–7

4.4. Results

  • Renderings preserve identity and remain consistent across views, with continuous control from γ=0 to the target exaggeration level.The visual comparison also reports fewer distortions than the baseline.Source: §4.4 · p. 7, Figure 5 · p. 7

4.5. Diffusion Based Editing

  • The diffusion-based editor produces plausible individual views but suffers geometry drift, unstable view-dependent specularities, and poor multi-view coherence.The authors evaluate both global edits and manually masked local edits.Source: §4.5 · p. 7

5. Ablations

The ablations show why caricature control needs both real-view and synthesized supervision, and why hair Gaussians are masked and frozen during synthesized supervision.

5.1. Alternated Training

  • Training only on input images fails to generalize to caricatured meshes, while training only on GT* distorts neutral renders; alternating both supervision types is necessary.A single shared Gaussian set is trained with real video frames and their caricatured GT* counterparts.Source: §5.1 · pp. 7–8
  • GaussianEditor’s global and local caricaturization edits show degraded geometry and appearance fidelity, especially in novel views.The two examples are a global edit and a local semantic edit.Source: Figure 6 · p. 8

5.2. Mask

  • Masking and freezing hair Gaussians during GT* supervision prevents the plastic-looking hair that appears when they train freely.GT* generation can misrepresent fine details, particularly hair.Source: §5.2 · p. 8

6. Limitations

The reported limitations include residual visual artifacts, imperfect eyelid geometry, and trade-offs in hair rendering.

6. Limitations

  • Residual specularity artifacts persist, and LAT over-stretching makes small eyelid inaccuracies visually noticeable.The authors also report imperfect eyelid closure in caricatured results.Source: §6 · p. 8
  • The deformed FLAME mesh does not span the full expression space, so eyes that should close completely may remain slightly open.Source: §6 · p. 8
  • Input-view-only hair supervision alleviates hair artifacts but can slightly over-smooth hair; masking and freezing hair Gaussians prevents GT* artifacts there.Source: §6 · p. 8, Figure 7 · p. 8

7. Discussion

The discussion emphasizes combining curvature-driven deformation with mesh-rigged 3DGS to maintain photorealism under large exaggerations. It also identifies recognition and diffusion-based editing as possible future directions.

7. Discussion

  • Curvature-driven deformation and mesh-rigged 3DGS form one controllable avatar model that remains photorealistic under large exaggerations.Source: §7 · p. 8
  • Alternating real views with LAT-generated caricature views and reliability masks lets one Gaussian set learn natural and caricatured appearance while retaining identity and expression.Source: §7 · p. 8
  • The authors hypothesize that using controllable exaggeration as face-recognition augmentation could improve robustness to pose and expression variability.Source: §7 · p. 8
  • They suggest that coupling geometry-grounded deformation with diffusion editors may support semantically guided joint control of shape and appearance.Source: §7 · p. 8

8. Implementation considerations

The implementation uses a shared training and evaluation protocol across subjects, with curvature-based caricaturization initialized once. It also specifies the CLIP configuration and inherited SurFhead settings.

8. Implementation considerations

  • Each subject’s 3DGS model is optimized for 120,000 iterations on one RTX 3090, taking approximately four hours of offline training.Source: §8 · p. 11
  • Experiments use 10 NeRSemble subjects, with EXP2 held out for testing and Camera 8 used for validation.Source: §8 · p. 11
  • Caricaturization solves the unconstrained Poisson equation once on the shared FLAME template using γ = 0.25, then reports metrics over 256 rendered test frames across camera views.Source: §8 · p. 11
  • Text-image alignment uses OpenAI CLIP ViT-B/32 with default preprocessing, while optimizer and Gaussian settings inherit SurFhead defaults unless specified otherwise.Source: §8 · p. 11
  • The CLIP comparison prompts contrast a naturally lit neutral head with a photorealistic caricature featuring an exaggerated nose and large ears.Source: §8 · p. 11

9. Linear Model and Error Analysis

The section approximates the curvature-weighted Poisson solution across exaggeration levels with a secant-based vertex blend. It analyzes how the weight approximation bounds the resulting geometric residual.

9. Linear Model and Error Analysis

  • The curvature magnitude is stabilized near K = 0 with ϵ, and Sγ denotes the weighted Poisson solution with Dirichlet boundary condition x* on ∂S.Source: §9 · p. 11
  • The secant interpolation is exact at γ = 0 and γ = γf; convexity makes its weight remainder non-negative, with the upper bound maximized at γf/2.Source: §9 · pp. 11–12
  • The secant-weight approximation enables a cheap vertex blend instead of solving a new weighted Poisson problem for every γ.The blend interpolates between S0 and Sγf using α = γ/γf and solves the Poisson equation with the secant weight.Source: §9 · pp. 11–12
  • The residual analysis freezes the surface operators and assumes a compact Riemannian surface with Lipschitz boundary and Dirichlet conditions.The intended error budget depends on domain ellipticity and Poincaré constants, the magnitude of ∇G S0, and the scalar secant remainder.Source: §9 · pp. 11–13
  • Poincaré and Lax–Milgram estimates bound the residual in the energy norm through a bounded functional and the difference between the blend and exact solution.Source: §9 · p. 12
  • The resulting energy bound depends on geometric constants of the domain and curvature evaluated at its global maximum.Source: §9 · p. 13

10. Caricature GT∗via one-shot stylization

The section evaluates one-shot stylization as an alternative for bridging the natural-caricature gap. It reports identity, expression, and cross-view consistency problems in the tested outputs.

10. Caricature GT∗via one-shot stylization

  • One-shot Deformable StyleGAN [46] outputs show identity–expression entanglement, degrading identity fidelity and expression accuracy.Source: §10 · p. 13
  • Under view changes or transferred expressions, outputs exhibit structural drift and collapse toward the reference style, limiting their suitability for 3DGS reconstruction.Source: §10 · p. 13
  • The evaluation uses the official [46] implementation with Style1, Style2, and Style3 exemplars and EMO3 and EMO4 expression prompts.Source: §10 · p. 13

11. Masking and GT∗

FLAME–image misregistration is amplified as caricature strength increases, especially around delicate facial geometry. Visibility-aware masking suppresses unreliable supervision to improve training stability.

11. Masking and GT∗

  • Misregistration in projected FLAME supervision grows with caricature strength, particularly around eyelids and eyeballs, while newly revealed triangles may lack reliable photometric support.Projection drift is most visible around delicate geometry; unfiltered drift can introduce erroneous supervision.Source: §11 · p. 13, Figure 9 · p. 14
  • The visibility-aware GT∗ mask suppresses newly visible triangles and masks fragile regions, including eyelids and ear tips, that are prone to amplified alignment error.Source: §11 · p. 13
  • Filtering inconsistent labels yields cleaner gradients and more stable appearance and geometry while retaining deformation-aware supervision and avoiding projection-drift and occlusion artifacts.Source: §11 · p. 14

12. Ablation: Alternating Supervision

Alternating supervision preserves fidelity at both the original and caricatured endpoints and supports smooth interpolation. Training on either domain alone overfits to that domain and produces characteristic failures.

12. Ablation: Alternating Supervision

  • The ablation compares original-only, GT∗-only, and alternating schedules under identical budgets, targeting γf=0.25 and evaluating γ values from 0 to 0.25.Source: §12 · p. 14
  • Original-only training distorts caricatured geometry, whereas GT∗-only training degrades at γ=0 and leaves holes or under-coverage in hair beyond tracked mesh support.Figures 10 and 11 illustrate the respective training schedules.Source: §12 · p. 14, Figure 10 · p. 15, Figure 11 · p. 16
  • Alternating supervision maintains high fidelity at both endpoints and smooth interpolation across γ, while avoiding the hair and occlusion failures of GT∗-only training.Figure 12 shows the alternating schedule.Source: §12 · p. 14, Figure 12 · p. 17

Related papers

Loading 2601.03319v1…