Source-linked AI summary
Counterfactual Route Optimization for Gaussian Head Avatar Modeling
Shikun Zhang, Yong Li, Yiqun Wang, Qiuhong Ke, Cunjian Chen
TL;DR
Head avatar objectives can favor different update directions at different training states, while existing schedules often prescribe their relative emphasis in advance. The paper evaluates alternative routes from the same state, factorizes their outcomes, and uses a lightweight controller to modulate training objectives. Experiments on NeRSemble demonstrate improved reconstruction quality on novel-view synthesis and reenactment tasks.
Problem
The relative benefit of geometry, appearance, and consistency objectives varies across training states, but existing pipelines often rely on fixed weights or handcrafted schedules.
Source: §1 · pp. 1–2Method
The framework evaluates short-horizon geometry, appearance, and joint routes from the same state, factorizes their realized utility, and trains a lightweight controller to apply bounded objective modulation.
Source: §1 · p. 2Results
Experiments on NeRSemble demonstrate improved reconstruction quality on novel-view synthesis and reenactment tasks.
Source: §5 · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
Head avatar modeling requires jointly optimizing multiple objectives with different dominant effects on geometry, appearance, and cross-view consistency. However, their relative effectiveness varies across training states, while existing pipelines typically rely on fixed loss weights or handcrafted stage-wise schedules. A central challenge is therefore to identify which optimization direction is more beneficial at each training state. We propose a counterfactual route optimization framework for Gaussian head avatar modeling, which characterizes state-dependent optimization preference from the realized effects of alternative updates rather than predefined heuristic weighting. Starting from the same training state, we perform short-horizon route-restricted lookahead over geometry, appearance, and joint update routes and evaluate their outcomes under a unified utility. The resulting counterfactual evidence is factorized into a geometry--appearance preference and a residual joint advantage, separately capturing the relative preference between individual update directions and the additional benefit of coordinated optimization. We further amortize this offline evidence into a lightweight controller that directly estimates the current optimization preference and applies bounded modulation to the training objectives during full avatar optimization. Experiments on the NeRSemble dataset validate the effectiveness of the proposed design, consistently outperforming existing methods while preserving clearer local facial structures and finer details.
1 INTRODUCTION
Avatar optimization must balance objectives whose relative benefit changes across training states, while existing schedules largely predetermine their emphasis. The paper estimates that preference from counterfactual update outcomes and uses it to modulate training objectives.
1 INTRODUCTION
- Avatar modeling has progressed from implicit neural fields to explicit point- and Gaussian-based representations and hybrid mesh–Gaussian formulations, improving fidelity, controllability, and efficiency.The introduction frames optimization organization as increasingly important as representation choices expand.Source: §1 · p. 1
- Optimization directions differ in their effects on geometry, appearance, fine detail, and cross-view consistency, and their relative benefit can change as training evolves.A direction effective at one stage may become less beneficial as reconstruction progresses.Source: §1 · p. 1
- Existing avatar methods largely predetermine objective emphasis through manually designed stage-wise schedules or coarse-to-fine curricula, rather than adapting it to each training state.The central challenge is identifying which direction is most beneficial for the current state.Source: §1 · pp. 1–2
- Adaptive methods often infer objective importance from losses, gradients, parameter contributions, or objective relationships rather than directly measuring alternative updates’ outcomes.A larger loss or stronger gradient does not necessarily mean that emphasizing that direction yields the greatest subsequent improvement.Source: §1 · p. 2
- The proposed framework evaluates alternative short-horizon optimization routes from the same state and factorizes their outcomes into geometry–appearance preference and additional joint-optimization benefit.This separates preference between individual directions from the added benefit of coordinated optimization.Source: §1 · p. 2
- A lightweight controller amortizes route evidence into bounded objective modulation during full training, with low online overhead while preserving the avatar representation and training framework.The approach is designed for dynamic coordination without changing the underlying reconstruction pipeline.Source: §1 · p. 2
2 RELATED WORK
Related work spans increasingly expressive avatar representations and increasingly structured optimization strategies. These approaches coordinate heterogeneous objectives, but their schedules or priorities are generally specified in advance rather than inferred from realized alternative updates.
2.1 ANIMATABLE HEAD AVATAR REPRESENTATIONS
- Avatar representations have moved from implicit neural fields through explicit Gaussian-based models to hybrid mesh–Gaussian formulations; the proposed hybrid uses an animatable FLAME face and 3D-Gaussian hair.The face and hair are jointly rendered as the final avatar.Source: §2.1 · p. 2, Figure 1 · p. 3
2.2 OPTIMIZATION STRATEGIES FOR AVATAR RECONSTRUCTION
- Avatar reconstruction methods use staged or progressive optimization, hierarchical refinement, and consistency constraints, but largely specify schedules, activation patterns, or objective emphasis in advance.Examples include TetGS, Fresco, ProgressiveAvatars, SVG-Head, and MVCHead.Source: §2.2 · p. 3
2.3 ADAPTIVE AND MULTI-OBJECTIVE OPTIMIZATION
- Adaptive optimization adjusts priorities using parameter contributions, task-specific updates, bi-level objectives, inter-task interactions, or multi-objective trade-offs.In head-avatar reconstruction, existing methods typically do not directly measure the effects of alternative update directions from the same state.Source: §2.3 · p. 3
3 METHOD
The method evaluates alternative optimization routes from a shared training state, factorizes their utility into directional preference and joint advantage, then amortizes that evidence into bounded objective modulation. Its avatar representation combines a parametric facial mesh with a deformable Gaussian hair field.
3 METHOD
- The framework evaluates geometry-, appearance-, and joint-oriented updates from the same state, factorizes their outcomes, and amortizes the evidence into a controller for full training.The alternative routes are short-horizon lookaheads; their temporary states are not retained in the main optimization trajectory.Source: §3 · p. 4, §3.2 · p. 5
3.1 AVATAR REPRESENTATION AND BASELINE OPTIMIZATION
- The avatar combines a posed parametric facial mesh with an anisotropic Gaussian hair field that deforms with the head state.The baseline training state summarizes the current solution under the underlying avatar framework.Source: §3.1 · p. 4
3.2 COUNTERFACTUAL ROUTE EVALUATION
- Route utilities compare normalized changes in current loss terms while penalizing structural degradation, using geometry, appearance, or their union as controlled update subsets.The parameter partition follows dominant roles, not strict geometry–appearance disentanglement; the joint utility also tests whether coordination adds value beyond either individual route.Source: §3.2 · pp. 4–5
3.3 FACTORIZED ROUTE EVIDENCE
- Factorized evidence separates continuous geometry–appearance preference from residual joint advantage over the stronger individual route.This distinguishes directional preference from the additional benefit of coordinated optimization rather than treating all three routes as equivalent choices.Source: §3.3 · p. 6
3.4 AMORTIZED ROUTE OPTIMIZATION
- A lightweight controller predicts factorized evidence from the current state, replacing online route rollouts with a single forward prediction and bounded objective corrections.The geometry–appearance factor sets relative emphasis, while the joint factor modulates shared optimization strength.Source: §3.4 · pp. 6–7
- Training objectives are grouped by dominant geometry, appearance, and consistency effects, with group-level cross-view consistency linking corresponding local observations across synchronized views.The grouping is a low-dimensional control abstraction, not an assertion that the effects are mutually exclusive.Source: §3.4 · pp. 6–7
4 EXPERIMENTS
Experiments evaluate avatars on NeRSemble using held-out sequences and views, comparing the method with prior approaches and testing route-modulation ablations. Results show advantages across novel-view synthesis and self-reenactment, while the ablation indicates that time-varying modulation adds value beyond fixed weighting.
4.1 EXPERIMENTAL SETUP
- Evaluation uses eight NeRSemble subjects, with held-out sequences and camera views for novel-view synthesis and self-reenactment.Cross-reenactment is evaluated qualitatively because paired ground truth is unavailable.Source: §4.1 · pp. 7–8
4.2 COMPARISONS WITH STATE-OF-THE-ART METHODS
- The method shows consistent performance advantages across novel-view synthesis and self-reenactment, with quantitative results relatively close to Fresco.Qualitative comparisons also show clearer local structures and finer details, including in cross-reenactment.Source: §4.2 · p. 8, Table 1 · p. 8, Figure 3 · p. 8
4.3 ABLATION STUDY
- The full model performs strongest, indicating that state-dependent adjustment adds value beyond a better fixed weighting configuration.Fixed-GC improves over baseline, and Mean Mod. further improves overall reconstruction quality; mouth artifacts decline progressively, with the full model recovering more complete teeth and cleaner lip boundaries.Source: §4.3 · p. 9, Figure 4 · p. 9, Table 2 · p. 9
5 CONCLUSION
The work frames head-avatar reconstruction as coordinating update directions as training evolves, using alternative update effects to guide state-dependent modulation. NeRSemble experiments show improved quality on novel-view synthesis and reenactment.
5 CONCLUSION
- The framework adapts optimization emphasis during training while preserving the avatar representation and reconstruction pipeline.It characterizes current optimization preference from the realized effects of alternative updates.Source: §5 · p. 9
- The reported experiments on NeRSemble demonstrate improved reconstruction quality for novel-view synthesis and reenactment.Source: §5 · p. 9
- Implementation, training, preprocessing, evaluation, ablation, and computational-cost details are documented in the supplementary material to facilitate reproduction.Source: REPRODUCIBILITY STATEMENT · p. 10
- The study uses publicly available NeRSemble data and reports no additional personal-data collection.Experiments are conducted for research purposes under the dataset’s established usage protocol.Source: ETHICS STATEMENT · p. 10
A IMPLEMENTATION DETAILS
The implementation uses specified training configurations and constructs route evidence offline from early training states. A lightweight controller is then fixed during avatar optimization and applies bounded modulation after a set iteration threshold.
Training Configuration
- MeGA, Fresco, and the proposed method run for 100 epochs per subject with batch size 16 and initial learning rate 1 × 10^-3.Experiments use one NVIDIA RTX A6000 GPU with 48 GB of memory; GaussianAvatars and TensorGA use 600K iterations under their official protocols.Source: A IMPLEMENTATION DETAILS · p. 13
Ablation Settings
- Fixed-GC uses consistency weight 0.002 and synchronized two-view supervision with a relative view-angle range of 25°–55° and a 40° fallback.Each sample uses 512 candidate anchors, 16 valid local groups, and a 41 × 41 patch; foreground coverage is at least 0.8 and the normal-visibility threshold is 0.2.Source: A IMPLEMENTATION DETAILS · p. 13
Counterfactual Evidence and Route Controller
- Offline labels use checkpoints from the first 10% of training, with geometry, appearance, and joint routes sharing a batch for one lookahead update.Route improvements are normalized component-wise, and the residual joint target is Qj − max(Qg, Qa).Source: A IMPLEMENTATION DETAILS · p. 13
- Online modulation begins after 20K iterations, with strength 0.1 and scaling factors clipped to [0.9, 1.1].The 30-dimensional-input controller has hidden dimension 64, remains fixed during avatar optimization, and is evaluated once per iteration without online lookahead.Source: A IMPLEMENTATION DETAILS · p. 13
B COMPUTATIONAL COST
The paper reports one-time offline costs for evidence generation and controller training, alongside inference timing. The controller operates during training and does not alter the rendering pipeline at inference.
B COMPUTATIONAL COST
- Offline evidence generation takes approximately 1 h 46 min per subject, while route-controller training takes about 3 minutes.Both stages are performed once before full avatar optimization.Source: B COMPUTATIONAL COST · p. 13
- Inference takes 57 ms per frame for the proposed method, compared with 55 ms for MeGA and Fresco.The route controller is used only during training and does not alter the inference rendering pipeline.Source: B COMPUTATIONAL COST · pp. 13–14, Table 3 · p. 14
- The local reconstruction-error visualization covers the mouth in novel-view synthesis and the eye in self-reenactment, with lower MAE indicating smaller errors.Source: Figure 6 · p. 14
C ADDITIONAL RESULTS AND ANALYSIS
The route-evidence ablation finds that factorizing geometry–appearance preference and joint modulation yields the strongest overall reconstruction quality. Additional visualizations report clearer local structures and lower reconstruction errors in novel-view synthesis and self-reenactment.
Novel-View Synthesis
- In novel-view synthesis, the method preserves sharper boundaries and more complete facial structures with fewer view-dependent artifacts.The reported improvements are especially around the mouth and teeth across unseen viewpoints.Source: C ADDITIONAL RESULTS AND ANALYSIS · p. 14
Local Error Visualization
- Local error maps show lower errors than displayed competitors in representative novel-view mouth and self-reenactment eye regions.Differences are particularly visible around lip boundaries, innermouth structures, and eye contours; lower mean absolute RGB error indicates closer local reconstruction.Source: C ADDITIONAL RESULTS AND ANALYSIS · p. 14
Route-Evidence Ablation
- The ablation compares geometry–appearance preference, joint modulation, their parallel-route variant, and the full factorized formulation.GA-only retains geometry–appearance preference; Joint-only predicts a joint scale; Parallel-Route treats all three route utilities as parallel preferences.Source: C ADDITIONAL RESULTS AND ANALYSIS · p. 14
- The full model achieves the strongest overall reconstruction quality, with best PSNR and LPIPS under both evaluation settings and comparable SSIM.Using either signal alone lowers overall performance; Parallel-Route improves over both single-signal variants but remains below the factorized formulation.Source: Table 4 · p. 15, C ADDITIONAL RESULTS AND ANALYSIS · p. 15
- Under self-reenactment, the full model alleviates lower-teeth artifacts and produces a cleaner local mouth structure than ablation variants.The qualitative comparison is shown in Figure 7.Source: C ADDITIONAL RESULTS AND ANALYSIS · p. 15, Figure 7 · p. 15