PaperlayerView exact revision on arXiv

Source-linked AI summary

Fresco++: Frequency-Guided and Canonical-Consistent Optimization for Fine-Grained Head Avatar Modeling

Shikun Zhang, Yong Li, Yiqun Wang, Qiuhong Ke, Cunjian Chen

arXiv:2610.11412v1cs.CV

TL;DR

Head-avatar optimization must recover fine appearance while maintaining coherent local structure across viewpoints. Fresco++ combines a coarse-to-fine frequency curriculum with canonical, reliability-aware multi-view consensus, and reports improved reconstruction and cross-view consistency across avatar representations.

  • Problem

    Head-avatar reconstruction must achieve fine visual fidelity, stable optimization, and coherent appearance across viewpoints despite coupled image-supervised components.

    Source: §I · p. 1

  • Method

    Fresco++ combines coarse-to-fine frequency optimization with Canonical Group Consensus, which associates canonical local structures and aggregates reliable multi-view observations into feature-level supervision.

    Source: §I · p. 2, C. Canonical Group Consensus · p. 5

  • Results

    Extensive experiments report improved reconstruction fidelity and cross-view coherence, with evaluations across avatar representations supporting Fresco++’s transferability.

    Source: §I · p. 2

  • Takeaways & Limitations

    The frequency curriculum and canonical consensus jointly support fine local detail and consistent corresponding regions across viewpoints.

    Source: §VI · p. 12

    Training adds moderate computation for frequency-aware filtering and multi-view feature consensus, while the original inference pipeline and runtime efficiency are essentially preserved.

    Source: §V · p. 12

Abstract

from arXiv · show

We propose Fresco++, a unified optimization framework for fine-grained and view-consistent head avatar reconstruction. Head avatar optimization is typically driven by per-view image supervision, which can lead to premature fitting of unstable high-frequency details and inconsistent local appearance across viewpoints. Fresco++ addresses these challenges by regulating both the progression of visual detail and the formation of cross-view supervision during optimization. For frequency-aware optimization, a progressive curriculum first stabilizes low-frequency structures and then introduces high-frequency constraints to recover fine facial and hair details without amplifying spurious responses at early stages. For cross-view optimization, we introduce Canonical Group Consensus, which associates local observations through shared canonical surface regions and establishes correspondence across different viewpoints. Geometric and visibility-aware screening removes unreliable observations, while the remaining multi-view evidence is aggregated in feature space to form a consensus target for supervising the current rendering. This design enforces local consistency without relying on a specific image-space parameterization and avoids additional rendering of the auxiliary view. Together, the frequency curriculum and canonical consensus provide stable optimization from coarse structures to fine details while maintaining coherent appearance across viewpoints. Extensive experiments on NeRSemble demonstrate improved reconstruction quality and cross-view consistency, while evaluations across diverse avatar representations further confirm the generality and transferability of Fresco++.

I. INTRODUCTION

Fresco++ targets the joint challenge of stable fine-detail reconstruction and coherent appearance across viewpoints. It combines a frequency curriculum with Canonical Group Consensus (CGC), and reports improved reconstruction fidelity and cross-view coherence across avatar representations.

I. INTRODUCTION

  • Fine-scale responses may be fitted before coarse structures are reliable, while local appearance or structure may drift across viewpoints.Existing strategies commonly treat optimization scheduling, multi-view supervision, and structural correspondence as separate mechanisms tied to a representation.Source: §I · p. 1
  • The frequency curriculum first emphasizes low-frequency structure, then progressively emphasizes high-frequency supervision for facial details, boundaries, and hair.This orders structural stabilization before detail refinement.Source: §I · pp. 1–2
  • CGC links corresponding local observations through shared canonical identities, screens unreliable evidence, and aggregates the remainder into feature-level supervision.This canonical formulation does not bind consistency to a particular image-space parameterization.Source: §I · p. 2
  • Fresco++ coordinates coarse-to-fine frequency progression with reliable cross-view supervision in a representation-transferable framework for head-avatar reconstruction.The framework removes Fresco’s dependence on a specific representation.Source: §I · p. 2
  • Extensive experiments report improved reconstruction fidelity and cross-view coherence, with evaluations across avatar representations and optimization and ablation analyses.The journal version extends Fresco, replacing UV-space consistency with canonical local-group correspondence and reliability-aware feature-level consensus.Source: §I · p. 2

II. RELATED WORK

Prior work spans diverse avatar representations and methods for optimization stability, cross-view consistency, and structural correspondence. The section identifies a remaining need to organize reliable multi-view supervision through structural correspondence that can generalize across representations.

A. Representations for Animatable Head Avatars

  • Head avatars use implicit, mesh, Gaussian, and hybrid representations, each offering different structures for geometry, appearance, deformation, or control.Hybrid approaches combine explicit geometry with flexible appearance primitives, increasing the importance of optimization methods that work across parameterizations.Source: A. Representations for Animatable Head… · p. 2

B. Optimization for Stable and Fine-Grained Reconstruction

  • Frequency-aware and coarse-to-fine methods schedule detail recovery, but do not necessarily establish whether cross-view observations correspond to the same local structure.Examples include frequency annealing and spectral regulation, as well as staged avatar optimization.Source: B. Optimization for Stable and… · pp. 2–3

C. Cross-View Consistency and Structural Correspondence

  • Cross-view methods use multi-view priors or consistency objectives, while structural correspondence methods use UV mapping, topological registration, or surface-local coordinates.The section identifies an underexplored formulation that uses stable structural correspondence to organize reliable multi-view supervision.Source: C. Cross-View Consistency and… · p. 3

III. METHOD

Fresco++ leaves the avatar’s underlying representation and rendering pipeline intact while optimizing both detail progression and cross-view supervision. Its overview combines a staged frequency branch with canonical-region consensus, grounded in synchronized views of a shared facial state.

III. METHOD

  • Fresco++ jointly regulates visual-detail progression and reliable cross-view supervision without redesigning the avatar representation.The frequency curriculum guides reconstruction from coarse structures toward fine details, while CGC associates canonical regions and consolidates reliable observations.Source: §III · p. 3

A. Preliminaries

  • The primary implementation combines a parametric FLAME facial surface with Gaussian primitives for hair and peripheral regions.The facial mesh is parameterized by identity shape, expression, and pose, and its canonical configuration is deformed into the posed surface.Source: A. Preliminaries · p. 3
  • Synchronized multi-view observations share the same underlying facial state and differ in camera viewpoint, allowing local structures to be related through a common canonical reference.The framework jointly renders the facial mesh and Gaussian field; the overview depicts low-to-high frequency refinement alongside screened canonical-group feature aggregation.Source: A. Preliminaries · p. 3, Figure 1 · p. 4

B. Spatially Localized Frequency Curriculum

The spatially localized frequency curriculum stabilizes coarse appearance before refining localized facial and hair details. Its staged schedule separates frequency objectives with different optimization roles.

B. Spatially Localized Frequency Curriculum

  • Fresco++ uses spatially localized frequency objectives to stabilize coarse structure before strongly optimizing fine details.The image-domain formulation preserves the spatial locations of frequency responses, unlike global spectral regulation.Source: B. Spatially Localized Frequency… · p. 4
  • Gaussian filtering extracts low-frequency components, whose loss emphasizes spatially coherent appearance and coarse structure.This provides a more stable basis before fine-scale details are strongly optimized.Source: B. Spatially Localized Frequency… · p. 4
  • A Difference-of-Gaussians operator extracts localized high-frequency responses for facial boundaries, wrinkles, and hair details.The method further weights structurally informative regions and uses a gradient-difference objective to preserve local gradient structure.Source: B. Spatially Localized Frequency… · pp. 4–5
  • A stage-wise schedule first consolidates low-frequency structure and then shifts toward high-frequency refinement, separating coarse structure formation from fine-detail recovery.Each stage maintains fixed frequency emphasis, avoiding competition between objectives with different optimization roles.Source: B. Spatially Localized Frequency… · p. 5

C. Canonical Group Consensus

Canonical Group Consensus (CGC) organizes cross-view supervision around shared canonical structures, screens observations for reliability, and forms a feature-level consensus target for the current rendering.

C. Canonical Group Consensus

  • CGC associates synchronized observations through canonical structural identities, filters unreliable evidence, and consolidates the remainder into a feature-level consensus target.Cross-view supervision is introduced only after structural correspondence and observation reliability have been established.Source: C. Canonical Group Consensus · p. 5
  • A persistent canonical identity defines correspondence before it is transferred to posed image observations, avoiding direct image-pixel correspondence.The same identity determines a common local structure projected into the two image planes.Source: C. Canonical Group Consensus · p. 5
  • A canonical group is a structural association of synchronized local observations sharing one canonical identity, not a particular primitive organization.Its realization can follow the avatar representation while the consensus formulation remains unchanged.Source: C. Canonical Group Consensus · p. 5
  • CGC retains a group only when its projected location is valid, its surrounding patch has sufficient foreground support, and the surface faces the camera.When depth observations are available, depth-inconsistent groups are also rejected; canonical correspondence alone does not establish trustworthy visual evidence.Source: C. Canonical Group Consensus · p. 5
  • For each retained group, CGC uses the current-view rendering and ground-truth patches from the current and synchronized auxiliary views to form feature-space supervision.A frozen feature extractor maps the patches into perceptual space, where the two ground-truth observations form the consensus representation.Source: C. Canonical Group Consensus · p. 6
  • The current rendered branch is optimized toward a fixed ground-truth consensus, without rendering the auxiliary viewpoint or propagating prediction errors between views.Only the rendered branch remains differentiable.Source: C. Canonical Group Consensus · p. 6

D. Joint Optimization

Joint optimization combines per-view reconstruction, progressive frequency control, and canonical cross-view constraints. The supplied qualitative comparison reports more faithful fine details under unseen viewpoints.

D. Joint Optimization

  • The joint objective combines reconstruction, frequency, and Canonical Group Consensus losses to optimize avatar parameters.The frequency loss follows a stage-wise curriculum, while λ_CGC controls the strength of cross-view consensus supervision.Source: D. Joint Optimization · p. 6
  • Reconstruction fits each view, frequency supervision progresses from stable low-frequency structure to high-frequency detail, and CGC constrains local regions with multi-view consensus evidence.Together, the objectives promote per-view reconstruction, progressive detail recovery, and cross-view structural consistency.Source: D. Joint Optimization · pp. 6–7
  • Under unseen viewpoints, Fresco++ produces more faithful fine details, particularly around teeth and facial wrinkles.The comparison also reports more stable local appearance across view changes.Source: Figure 3 · p. 7

IV. EXPERIMENTS

Experiments evaluate Fresco++ on NeRSemble across novel-view synthesis, self-reenactment, and cross-reenactment, using shared training and evaluation splits. The setup also specifies the frequency schedule and the reliability screening used for CGC.

Dataset and evaluation

  • Eight NeRSemble subjects are evaluated, with nine predefined sequences and 15 camera views used for training and one sequence and the remaining view held out.Cross-reenactment transfers expressions from another subject and is evaluated qualitatively because paired ground-truth images are unavailable.Source: §IV · p. 7

Evaluation settings and metrics

  • Novel-view synthesis and self-reenactment are scored with PSNR, SSIM, and LPIPS within a consistent foreground region; cross-reenactment is assessed qualitatively.Higher PSNR and SSIM and lower LPIPS indicate better reconstruction quality.Source: §IV · pp. 7–8

Implementation details

  • Fresco++ activates low-frequency supervision and CGC from epochs 10–50, then introduces high-frequency supervision from epochs 50–80 after warm-up.The reported weights are λLF = 0.1, λCGC = 0.002, and λHF = 0.005.Source: §IV · p. 8
  • CGC screens observations by foreground validity, surface visibility, and depth-based occlusion, and uses a frozen VGG-16 feature extractor.It is used only during training and adds no inference-time computation.Source: §IV · p. 8

A. Comparisons with State-of-the-Art Methods

Fresco++ outperforms representative methods on novel-view synthesis and self-reenactment, with qualitative comparisons also showing improved local detail and expression transfer. Its novel-view result includes a 0.68 dB PSNR gain over GaussianAvatars.

Quantitative comparison

  • 32.18 dB PSNR on novel-view synthesis gives Fresco++ the best result on all three metrics and a 0.68 dB PSNR gain over GaussianAvatars.Table I compares previous methods on novel-view synthesis and self-reenactment.Source: A. Comparisons with State-of-the-Art… · p. 8
  • On self-reenactment, Fresco++ has the highest PSNR and lowest LPIPS, and shares the best SSIM with TensorGA.The authors report consistent improvement over Fresco on both novel-view synthesis and self-reenactment.Source: A. Comparisons with State-of-the-Art… · p. 8

Qualitative comparison

  • Fresco++ preserves fine details such as earrings and reproduces expression-dependent eye opening and cheek wrinkles more faithfully during self-reenactment.The reported visual comparison concerns unseen facial motions.Source: A. Comparisons with State-of-the-Art… · p. 8
  • Under unseen viewpoints, Fresco++ better preserves details such as teeth and facial wrinkles than methods that show over-smoothed textures or unstable local appearance.Local error comparisons also report lower errors around the mouth and eyes in both self-reenactment and novel-view synthesis.Source: A. Comparisons with State-of-the-Art… · p. 8
  • In cross-reenactment, Fresco++ better preserves target identity while transferring expressions, including mouth deformation and strong eye-region contraction.The captioned comparison highlights local appearance under cross-identity motion transfer.Source: A. Comparisons with State-of-the-Art… · p. 8

B. Ablation Studies

Ablations show that CGC improves cross-view supervision over UV-space consistency, while combining CGC with the frequency curriculum yields the strongest reported reconstruction results. Qualitative comparisons show more stable local appearance across viewpoints with CGC.

Component analysis

  • On novel-view synthesis, CGC raises average PSNR from 30.79 dB to 31.50 dB and lowers LPIPS from 0.066 to 0.060, outperforming UV-space consistency across all reported metrics.The UV-space variant reports 31.17 dB PSNR and 0.064 LPIPS; the ablation averages results over evaluated subjects.Source: B. Ablation Studies · p. 8, Table II · p. 9
  • CGC also improves self-reenactment reconstruction fidelity and perceptual quality over the UV-based formulation.The reported results connect canonical structural correspondence with more effective cross-view supervision than UV-space consistency.Source: B. Ablation Studies · p. 9
  • 32.18 dB PSNR and 0.056 LPIPS on novel-view synthesis, and 30.61 dB PSNR and 0.063 LPIPS on self-reenactment, are the complete Fresco++ model’s best results across both settings.The authors attribute the additional improvement over CGC alone to complementary benefits from canonical consistency and coarse-to-fine frequency optimization.Source: B. Ablation Studies · p. 9, Table II · p. 9

Qualitative analysis of cross-view consistency

  • Across viewpoints, CGC preserves more stable local structures, clearer boundaries, and more consistent fine details than the baseline or UV-space consistency.The comparison highlights regions around the hairline, temporal region, and ear boundary.Source: B. Ablation Studies · p. 10, Figure 6 · p. 10
  • CGC forms consistency supervision by associating shared canonical structural identities and using reliable multi-view observations.This provides constraints on both individual-view reconstruction quality and the stability of corresponding local structures across viewpoints.Source: B. Ablation Studies · p. 10

C. Analysis of Frequency Curriculum

Frequency-order ablations favor low-to-high scheduling: LF→HF performs best overall and has lower later-stage loss variation than the alternatives. Early low-frequency supervision also produces cleaner intermediate reconstructions than early high-frequency supervision.

Effect of frequency scheduling

  • The schedule variants differ only in frequency order: LF→HF stages low then high frequencies, HF→LF reverses them, and LF+HF activates both objectives through the guided stages.The variants share the same optimization settings and final stage without frequency supervision.Source: C. Analysis of Frequency Curriculum · p. 10
  • LF→HF achieves the strongest overall schedule results, including the best self-reenactment results and highest novel-view PSNR and SSIM.Staged frequency optimization consistently outperforms simultaneous low- and high-frequency supervision in the reported comparison.Source: C. Analysis of Frequency Curriculum · p. 10, Table III · p. 10

Optimization stability

  • After the early training stage, LF→HF has lower cumulative variation in both head RGB and SSIM losses than HF→LF or simultaneous frequency optimization.Lower cumulative variation indicates smaller changes in the reconstruction trajectory.Source: C. Analysis of Frequency Curriculum · p. 11, Figure 7 · p. 11

Effect of early frequency supervision

  • At epoch 30, early high-frequency supervision tends to introduce spurious local details, while early low-frequency supervision produces cleaner, more coherent reconstructions.The reported artifacts appear particularly around the forehead and mouth.Source: C. Analysis of Frequency Curriculum · p. 11, Figure 8 · p. 11

D. Transferability across Avatar Representations

Fresco++ transfers to GaussianAvatars and TensorGA, improving reconstruction metrics and local detail while adding training computation but essentially preserving inference efficiency.

D. Transferability across Avatar Representations

  • GaussianAvatars recovers mouth-corner wrinkles and facial textures more faithfully with fewer eye-region artifacts, while TensorGA improves mouth textures and teeth reconstruction.These qualitative comparisons are shown in Fig. 9 and Fig. 10.Source: Figure 9 · p. 11, D. Transferability across Avatar… · p. 12, Figure 10 · p. 12
  • Across GaussianAvatars and TensorGA, Fresco++ improves PSNR and SSIM and reduces LPIPS for novel-view synthesis; self-reenactment PSNR and LPIPS improve while SSIM remains stable.The reported pattern supports transfer across both purely Gaussian representations.Source: D. Transferability across Avatar… · p. 11, Table IV · p. 11
  • Fresco++ transfers to two purely Gaussian models by applying the frequency curriculum to rendered outputs and adapting canonical-group construction to each model’s structural associations.The evaluation compares GaussianAvatars and TensorGA with and without Fresco++ under novel-view synthesis and self-reenactment.Source: D. Transferability across Avatar… · p. 11
  • Transfer adds training time and only a small peak-memory increase, while inference time remains essentially unchanged.The reported overhead mainly comes from image-space filtering and CGC’s multi-view feature-consistency computation.Source: D. Transferability across Avatar… · p. 12, Table V · p. 12

V. LIMITATIONS AND FUTURE WORK

Fresco++ concludes that its frequency curriculum and Canonical Group Consensus improve detail fidelity and cross-view stability, while identifying head-avatar scope and training computation as limitations.

V. LIMITATIONS AND FUTURE WORK

  • The current evaluation mainly covers head-avatar reconstruction; extending the strategy to broader dynamic human representations and more diverse scenarios remains future work.Source: §V · p. 12
  • Fresco++ adds moderate training-time computation from frequency-aware filtering and multi-view feature consensus, while preserving the original inference pipeline and runtime efficiency.The paper proposes more efficient training-time consistency supervision as future work.Source: §V · p. 12

VI. CONCLUSION

  • The frequency curriculum and Canonical Group Consensus together improve local-detail fidelity and cross-view stability, with transfer studies across Gaussian avatar representations supporting the strategy’s broader applicability.The conclusion notes additional training cost but essentially unchanged inference efficiency.Source: §VI · p. 12

Related papers

Loading 2610.11412v1…