PaperlayerView exact revision on arXiv

Source-linked AI summary

One Basis to Animate Them All: Gaussian Blendshape Distillation for Real-Time Avatars

Ramazan Fazylov, Stamatis Lefkimmiatis, Ivan Laptev

arXiv:2610.02207v1cs.CVcs.AIcs.HCcs.LG

TL;DR

Per-frame neural animation can dominate inference cost and limit CPU or mobile deployment. GALA replaces that decoding with a shallow coefficient predictor and shared linear blendshape basis; across three avatar models, it closely approaches host-model quality while reducing CPU animation cost by up to three orders of magnitude and enabling 60 fps mobile animation.

  • Problem

    Per-frame neural networks update Gaussian attributes from expression or pose parameters, and this animation computation can dominate inference cost and limit CPU or mobile deployment.

    Source: §1 · p. 2

  • Method

    GALA distills pretrained Gaussian-avatar animation into a rendering-aware, block-local blendshape basis under a memory budget, controlled by a shallow coefficient predictor without retraining the host model.

    Source: §1 · p. 2

  • Results

    Across three avatar models, GALA closely preserves host-model animation quality, accelerates CPU execution by up to three orders of magnitude, and enables 60 fps mobile animation.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The shared basis captures learned avatar deformations and supports animation of identities excluded from basis construction.

    Source: §1 · p. 2

    At a 48 MB budget, predicted coefficients have PSNR 5.7 dB lower for AGORA and 12.3 dB lower for FlexAvatar than projected coefficients.

    Source: §5 · p. 9

Abstract

from arXiv · show

3D Gaussian avatars support fast rendering, however, their real-time animation is often challenged by the costly neural inference. We address this bottleneck and show that the animation of pretrained avatar models can be closely approximated by a linear combination of identity-independent blendshapes. Building on this finding, we introduce GALA (Gaussian Animation via Linear Approximation), a distillation method that replaces per-frame heavy neural decoding with a shallow coefficient predictor and a linear blend. To improve fidelity and reduce memory requirements, we propose to construct the basis using block-local PCA under a rendering-aware metric and a memory budget. Our method learns a shallow MLP network to predict blendshape coefficients and applies to various animation architectures without retraining original models. We validate GALA by accelerating the inference of three distinct avatar models for 3D animation of facial expressions and full-bodies with clothing dynamics. Across these models, our distillation generalizes to held-out identities and reduces CPU animation cost by up to three orders of magnitude while preserving most of the rendering quality. Excellent results of our method confirm the shared linear structure of learned avatar representations and enable highly efficient and accurate animation at frame rates reaching up to 60fps on mobile devices. Project page: https://ramazan793.github.io/gala/

1 INTRODUCTION

GALA distills pretrained Gaussian-avatar animation into a shared linear blendshape basis controlled by a shallow network, targeting costly per-frame neural computation. Across three avatar models, it closely approaches host-model animation accuracy, speeds CPU animation by up to three orders of magnitude, and enables 60 fps mobile animation.

1 INTRODUCTION

  • Efficient 3D Gaussian rendering does not ensure efficient animation: per-frame neural updates to Gaussian attributes can dominate inference and limit CPU or mobile deployment.The updates take expression or pose parameters as input.Source: §1 · pp. 1–2
  • A shared identity-independent blendshape basis closely approximates pretrained avatar animation and generalizes to identities excluded from basis construction.Identity-dependent motion is retained through coefficients while the deformation basis is shared.Source: §1 · pp. 2–3
  • Unlike subject-specific fitting or predefined rigs, GALA extracts a reusable basis from a trained model representing many identities.The basis approximates deformations learned by the original network and can be reused for new subjects without reconstruction.Source: §1 · p. 2
  • GALA uses rendering-aware, block-local PCA under a memory budget and a shallow coefficient predictor, then adds a linear blend to a cached neutral avatar without retraining the host.The basis prioritizes deformations relevant to visual fidelity and allocates components by estimated image-space error reduction per stored byte.Source: §1 · p. 2
  • GALA closely approaches host-model animation accuracy, speeds CPU animation by up to three orders of magnitude, and enables 60 fps mobile animation across three avatar models.The evaluated hosts cover head generation, one-shot head reconstruction, and full-body reconstruction with clothing dynamics.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans neural avatar generation and reconstruction, linear face and body models, and photorealistic Gaussian blendshapes. GALA’s overview places its approach in this landscape as a shared-basis distillation pipeline.

2 RELATED WORK

  • Avatar research covers GAN-based head generation, one-shot reconstruction, per-subject video fitting, and full-body animation with pose-dependent clothing dynamics (Chan et al., 2022; Deng et al., 2024a;b; Qian et al., 2024; Kwon et al., 2026).The surveyed approaches include conditioning generation on morphable-model parameters, reconstructing from a single image, and articulating full-body avatars.Source: §2 · p. 3
  • Linear face and body models range from PCA and morphable models to blendshapes and multilinear models, with FLAME and SMPL-X providing driving parameters for the studied avatars (Pavlakos et al., 2019).The paper’s distinction is that its linear approximation is extracted from neural animation outputs rather than fitted directly to scans.Source: §2 · p. 3
  • Photorealistic Gaussian blendshape methods typically fit subject-specific bases, while fixed-rig methods use predefined articulation; GALA extracts a shared basis reusable for unseen identities (Ma et al., 2024; Dhamo et al., 2024; He et al., 2025; Zielonka et al., 2025).The basis captures deformations learned by a pretrained neural model.Source: §2 · p. 3
  • Figure 2 summarizes GALA’s pipeline: residuals undergo spatially local, rendering-aware PCA under a memory budget, then a shallow predictor estimates coefficients for linear blending.The neutral avatar is computed once per subject; each frame uses one predictor evaluation and a linear blend.Source: Figure 2 · p. 4

3 METHOD

GALA approximates pretrained avatar animation with a neutral avatar and a linear basis, replacing per-frame neural decoding with predicted blendshape coefficients. Its basis construction uses PCA organized by attribute group and spatial block, weighted for rendering quality and allocated under a memory budget.

3.1 PRELIMINARIES

  • Host models map a fixed identity descriptor and animation input to Gaussian attributes, requiring consistent Gaussian correspondence across identities and poses.The driving input may include motion history for temporally conditioned models.Source: §3.1 · p. 4

3.2 GAUSSIAN BLENDSHAPE DISTILLATION

  • GALA approximates each animated avatar as its neutral representation plus a linear basis multiplied by coefficients conditioned on identity and driving signals.The neutral representation is estimated once per animation sequence.Source: §3.2 · p. 4

3.2.1 SIMPLE PCA BASIS

  • Uncentered PCA applies truncated SVD to neutral-relative residuals, with projected coefficients and zero coefficients recovering the exact neutral avatar.Residuals are stacked across identities and animation samples to form the matrix used to extract a rank-K basis.Source: §3.2.1 · pp. 4–5
  • Joint residuals encode shared animation and can transfer to held-out identities with consistent Gaussian correspondence; simple PCA mixes differently scaled attributes, spends memory on global components, and ignores rendering quality.These limitations motivate separating attributes and spatial regions and using a rendering-aware metric.Source: §3.2.1 · p. 5

3.2.2 IMPROVED PCA BASIS

  • The improved basis separates Gaussian attributes and partitions Gaussians into spatial blocks, storing each local component only for its block.Separate attribute bases allow representation capacity to be selected independently for groups such as position, scale, rotation, opacity, and appearance.Source: §3.2.2 · p. 5
  • Rendering-aware PCA weights attribute errors by their effect on rendered images, while a ridge term retains weakly observed directions and an optional landmark term adds geometric supervision.The metric is estimated using per-Gaussian diagonal blocks rather than constructing the full rendering matrix.Source: §3.2.2 · p. 5
  • Components are allocated jointly across attribute groups and spatial blocks by estimated image-error reduction per stored byte until the memory budget is exhausted.The component value is derived from metric-weighted PCA variation and is made comparable across groups by undoing normalization and the rendering weight.Source: §3.2.2 · pp. 5–6

3.2.3 REGRESSING BLENDSHAPE COEFFICIENTS

  • A shallow MLP predicts fixed-basis coefficients from identity and animation inputs, avoiding host-network evaluation to obtain projected coefficients at inference.Identity conditioning allows the same expression or pose to produce subject-specific deformations despite a shared basis.Source: §3.2.3 · p. 6

4 EXPERIMENTS

Experiments test GALA on three avatar hosts, finding that it largely preserves host-model quality while reducing CPU animation time. Ablations examine basis construction, component allocation, spatial blocks, and alternative distillation.

4.1 EXPERIMENTAL SETUP

  • AGORA, FlexAvatar, and DynaAvatar cover GAN head generation, one-shot head reconstruction, and clothed full-body reconstruction; results are reported on identities excluded from training data.FlexAvatar evaluation also follows its per-subject enrollment protocol, fine-tuning the coefficient network on host-model outputs.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 RESULTS

  • All three distilled models closely follow their hosts and outperform competing methods on most metrics across their respective benchmarks.Table 2 reports benchmark comparisons for AGORA, FlexAvatar, and DynaAvatar.Source: Table 2 · p. 7, §4.2 · p. 7
  • 154→5.5 ms, 234→4.4 ms, and 42.9 s→6.1 ms (16.1 ms with skinning) are the respective CPU animation-step reductions.The distilled models reach 60 frames per second on a mobile phone, compared with AGORA’s 1 frame per second there and DynaAvatar’s 0.3 frames per second on a desktop GPU.Source: §4.2 · p. 7
  • Predicted-coefficient errors remain small and concentrate at thin structures and silhouette outlines; projected coefficients nearly reproduce hosts but require running the host model.Examples cover facial expressions and body motion, with error maps showing absolute pixel differences from the host.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

4.3 ABLATION STUDY

  • Simple PCA trails the full method by 9.7 dB for AGORA, 11.5 dB for FlexAvatar, and 13.2 dB for DynaAvatar.Joint bases cost 4.7, 3.6, and 8.8 dB; explained-variance scoring costs 5.3, 2.7, and 13.3 dB; rendering-aware PCA adds 0.25, 0.35, and 0.58 dB over Euclidean PCA, respectively.Source: Table 3 · p. 8, §4.3 · p. 8
  • Against one shared rank, PSNR falls by 6.2, 1.2, and 8.5 dB; per-group-and-block allocation adds 3.4, 0.1, and 0.7 dB over per-group ranks.The allocation comparison holds bases and memory budget fixed while changing rank constraints.Source: Table 4 · p. 8, §4.3 · pp. 8–9
  • Across all three hosts and tested memory budgets, thinner and shallower student networks do not approach GALA in speed and accuracy.Figure 4 compares quality against CPU animation-step time, with lower values better on both axes.Source: Figure 4 · p. 9, §4.3 · p. 9
  • At a fixed basis size, 32 local blocks improve PSNR over one global block by 8.2, 6.9, and 8.6 dB for AGORA, FlexAvatar, and DynaAvatar.Larger block counts showed no consistent improvement with predicted coefficients, as more components make coefficients harder to regress.Source: §4.3 · p. 9

5 LIMITATIONS AND FUTURE WORK

GALA’s coefficient prediction remains a limitation, and its fixed spatial partition may not be optimal for every motion. The method also inherits limitations of its host models.

5 LIMITATIONS AND FUTURE WORK

  • At 48 MB, predicted-coefficient PSNR trails projected-coefficient PSNR by 5.7 dB for AGORA and 12.3 dB for FlexAvatar.A larger budget mainly improves results with projected coefficients.Source: §5 · p. 9
  • The spatial partition is fixed at 32 blocks for all hosts, although an optimal partition may depend on motion and component counts.The paper leaves jointly selecting the partition and component numbers for future work.Source: §5 · p. 9
  • GALA inherits the limitations of its host models; the authors leave training avatar models directly with an identity-shared blendshape basis for future work.Source: §5 · p. 9

6 CONCLUSION

GALA replaces per-frame neural decoding with a shallow coefficient network and a shared blendshape basis, closely preserving quality while improving computational performance. The appendices provide implementation, evaluation, comparison, and qualitative details.

6 CONCLUSION

  • GALA replaces per-frame neural decoding with a shallow coefficient network and a linear blendshape combination, using a basis shared across identities.The conclusion contrasts this with prior Gaussian blendshape methods that require a separate basis for each subject.Source: §6 · p. 9
  • GALA preserves the original models’ quality while improving computational performance, accelerating CPU execution by up to three orders of magnitude and enabling 60 fps on a mobile phone.The method requires no retraining of the original models.Source: §6 · p. 10

APPENDIX

  • The appendices cover metric and basis construction, host models, implementation, coefficient networks, evaluations, comparisons, and qualitative results.They also visualize individual blendshape motions.Source: APPENDIX · p. 14

A RENDERING-AWARE METRIC AND BASIS CONSTRUCTION

The appendix details how GALA estimates a rendering-aware metric, constructs and allocates local PCA bases, and applies the method across three host models. It also describes the browser demos’ per-frame animation pipeline.

A RENDERING-AWARE METRIC AND BASIS CONSTRUCTION

  • The rendering-aware metric approximates the Gauss–Newton Hessian with per-Gaussian blocks, retaining within-Gaussian channel interactions while discarding interactions across Gaussians and attribute groups.The blocks are estimated with stochastic probes and normalized separately for each attribute group; the landmark metric uses the Jacobian of landmark positions.Source: A RENDERING-AWARE METRIC AND BASIS… · p. 14
  • Block-local PCA transforms residuals into the metric space and uses uncentered PCA to obtain bases and projected coefficients.The metric is block diagonal by Gaussian; setting β = γ = 0 reduces construction to simple PCA applied to each block.Source: A RENDERING-AWARE METRIC AND BASIS… · p. 14
  • Components are allocated greedily by estimated error reduction per byte, combining image error with weighted attribute error and, when γ > 0, landmark error.Components are retained in spectral order within each local basis.Source: A RENDERING-AWARE METRIC AND BASIS… · p. 14

B HOST MODELS

  • The three hosts span head generation, single-image head reconstruction, and full-body reconstruction with clothing dynamics, using distinct animation architectures.AGORA uses a per-frame deformation branch; FlexAvatar regenerates Gaussian attributes with its decoder; DynaAvatar uses a pose-history-conditioned dynamic transformer.Source: B HOST MODELS · p. 15

C MOBILE BROWSER DEMO

  • In the browser demo, the coefficient network predicts blendshape weights on the GPU, then a second pass forms Gaussian attributes before sorting and rasterization.The neutral avatar and basis load once, and no host network runs in the browser.Source: C MOBILE BROWSER DEMO · p. 15

D IMPLEMENTATION DETAILS

Implementation uses spatially blocked, rendering-aware bases under a storage budget and adapts enrollment and timing protocols to the host models. The coefficient networks take identity and driving inputs, with a temporal architecture for DynaAvatar.

D IMPLEMENTATION DETAILS

  • The basis uses 32 spatial blocks, a rendering-aware metric, and a memory budget; landmark weighting is used only for FlexAvatar positions.The metric uses β = 16, or β = 64 for DynaAvatar; bases use half precision for the head models and single precision for DynaAvatar.Source: D IMPLEMENTATION DETAILS · p. 15
  • The mobile demos show AGORA driven by tracked expressions, five FlexAvatar avatars reconstructed from photos, and five DynaAvatar full-body avatars with clothing dynamics.The figure overlays frame rate and animation-step timing.Source: Figure 5 · p. 16
  • FlexAvatar enrollment estimates a constant neutral-avatar correction and fine-tunes the coefficient network for 200 steps using host outputs, without target frames or rendering.The protocol begins from the identity representation fitted to the test subject’s input images.Source: D IMPLEMENTATION DETAILS · p. 16
  • Animation-step timing measures CPU computation of Gaussian attributes without rendering on an Intel Core Ultra 9 285K using eight threads and batch size one.For DynaAvatar, the 10 ms per-frame linear blend skinning cost is excluded from Table 5 timings and included in Figure 4.Source: D IMPLEMENTATION DETAILS · pp. 16–17, Table 5 · p. 16

E COEFFICIENT NETWORKS

  • The coefficient network receives identity descriptors and host-specific driving signals; DynaAvatar processes the current and 15 preceding pose frames with temporal convolutions.Its output is anchored at a reference pose so that this pose yields the neutral avatar for each subject.Source: E COEFFICIENT NETWORKS · p. 17
  • Training predicts projected residual coefficients using identity and animation inputs paired across subjects, and selects checkpoints on held-out identities.The loss is normalized per attribute group by that group’s coefficient energy on the training data.Source: E COEFFICIENT NETWORKS · pp. 17–18

F EVALUATION DETAILS

Evaluation uses each host model’s benchmark and metrics, with specified test protocols for DynaAvatar and FlexAvatar. The reported ablation finds that the rendering-aware method improves most quality metrics over simple PCA.

F EVALUATION DETAILS

  • AGORA uses FFHQ metrics for image quality, expression and jaw animation, identity consistency, and head pose; DynaAvatar uses PSNR, SSIM, and LPIPS on 4D-Dress.FlexAvatar is evaluated on VFHQ portrait animation and Ava256 avatar creation, using image-quality, animation-accuracy, and identity-preservation metrics.Source: F EVALUATION DETAILS · p. 18
  • DynaAvatar evaluation follows the 4D-Dress test protocol on 83 sequences and 12,200 frames, with host and distilled model assessed on the same renders.The measured host results are 23.86 dB PSNR, 0.959 SSIM, and 0.063 LPIPS, reproducing the published numbers within 0.12 dB.Source: F EVALUATION DETAILS · p. 18
  • For Ava256, the authors’ protocol omits the specific subjects, sequences, and expression frames, so evaluation uses eight unseen subjects and the EXP free face sequence.The host and GALA model use the same identities, inputs, and target frames; the single-shot host PSNR is 19.27 dB over eight subjects versus 16.9 dB over five in the cited paper.Source: F EVALUATION DETAILS · p. 18
  • 2.4 dB PSNR for AGORA, 0.9 dB for FlexAvatar, and 0.6 dB for DynaAvatar: the full method beats simple PCA on every quality metric except identity metrics.Higher ID and CSIM for simple PCA indicate under-animation; a second allocation rule chooses attribute-group budget splits without rendering.Source: F EVALUATION DETAILS · p. 18

G COMPARISON TO ALTERNATIVE DISTILLATION

GALA compares favorably with compact distilled networks on speed and accuracy, and with per-subject blendshape avatars on cross reenactment across the tested datasets. Per-subject fits nevertheless score better on self reenactment.

G COMPARISON TO ALTERNATIVE DISTILLATION

  • Distilled baselines are reduced copies of each host’s animation network, trained on frozen-host outputs; DynaAvatar students also attend to sampled Gaussians and interpolate to all 40,000.The comparison spans student sizes, CPU times, and quality.Source: G COMPARISON TO ALTERNATIVE DISTILLATION · pp. 18–19, Table 8 · p. 19
  • The best distilled AGORA network reaches FID 3.60 in 124 ms, versus 3.47 in 5.5 ms for GALA; no distilled network is both faster and more accurate than GALA.For FlexAvatar, only the largest transformer approaches host LPIPS, at 175 ms—40 times slower than GALA.Source: G COMPARISON TO ALTERNATIVE DISTILLATION · p. 19

H COMPARISON TO PER-SUBJECT BLENDSHAPE AVATARS

I ADDITIONAL QUALITATIVE RESULTS

The difficult examples include held-out identities, few-shot and single-shot subjects, and a DynaAvatar frame with the sequence’s largest error; one Ava256 example exposes a large coefficient-prediction failure.

I ADDITIONAL QUALITATIVE RESULTS

  • Rows 1 and 2 show held-out AGORA identities at frames with the host model’s largest residual.The figure compares the neutral avatar, projected coefficients, and GALA-predicted coefficients.Source: Figure 9 · p. 24
  • In Ava256 row 4, GALA’s predicted coefficients yield 23.6 dB PSNR against the host model, versus 40.2 dB with projected coefficients.Rows 3 and 4 show held-out Ava256 subjects in few-shot and single-shot settings.Source: Figure 9 · p. 24
  • Row 5 shows the DynaAvatar sequence frame with the largest error for GALA.Figure 9 presents these as more difficult examples alongside the coefficient-network failure case.Source: Figure 9 · p. 24, I ADDITIONAL QUALITATIVE RESULTS · p. 24

J VISUALIZATION OF BLENDSHAPES

The basis captures localized motions that remain consistent across identities, with examples spanning facial features and garment movement. The visualizations show conditional activations rather than isolated basis vectors.

J VISUALIZATION OF BLENDSHAPES

  • Local blendshapes produce consistent motions across identities, including eye, mouth, and garment motions across AGORA, FlexAvatar, and DynaAvatar.The additional examples include a smile, a brow-and-eyelid motion, and the opposite front part of the garment.Source: J VISUALIZATION OF BLENDSHAPES · p. 24, Figure 10 · p. 25, Figure 11 · p. 25
  • The displayed states are conditional activations of basis parts, not isolated vectors; head-model states use coefficient patterns near the 2nd and 98th percentiles.For DynaAvatar, one component is set to an extreme percentile while other coefficients come from nearby training vectors.Source: J VISUALIZATION OF BLENDSHAPES · pp. 24–25
  • AGORA and FlexAvatar examples use held-out identities, whereas the DynaAvatar subjects belong to the training data.The bases shown are 154.1, 48, and 88.3 MB, respectively.Source: J VISUALIZATION OF BLENDSHAPES · p. 25
Loading 2610.02207v1…