PaperlayerView exact revision on arXiv

Source-linked AI summary

Shape-Bayes: Bayesian Inference of Structured Shapes under Visual Ambiguity

Mani Kumar Tellamekala, Tosh Brown, Michel Valstar

arXiv:2610.09032v1cs.CV

TL;DR

Shape inference from pixels is ambiguous, especially when occlusion removes visual evidence, while shape regression commonly relies on deterministic point estimates. SHAPE-BAYES balances uncertain observations with an adaptive geometric prior to infer shape posteriors, improving structural integrity, accuracy, and uncertainty calibration in face regression.

  • Problem

    Shape inference is ambiguous when visual features are occluded, and deterministic shape regression does not represent a distribution of plausible shapes.

    Source: §1 · p. 1

  • Method

    SHAPE-BAYES combines distilled aleatoric uncertainty with a learned shape prior and a Bayesian update to infer a posterior over shapes.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    On 300W under dynamic occlusion, SHAPE-BAYES reduces HR-noSA relative NMEocc by 12.4% and relative NMEall by 6.5%, while reported results show improved structural integrity and calibration.

    Source: §4 · pp. 6–8

  • Takeaways & Limitations

    Across six face-alignment architectures, SHAPE-BAYES consistently reduces shape error, sharpens uncertainty calibration, and enforces structural validity.

    Source: §6 · p. 10

Abstract

from arXiv · show

Perceiving structured shapes, such as human faces, from pixels is an inherently ambiguous task in real-world conditions. Yet, shape inference is largely posed as a deterministic regression task predicting fixed spatial coordinates. We find that deterministic regression is brittle when visual evidence is ambiguous or incomplete; under severe occlusions deterministic models exhibit structural collapse, predicting incoherent shapes or reverting to generic averages. To address this, we introduce Shape-Bayes, a probabilistic framework that couples uncertainty-aware visual perception with Bayesian shape reasoning. Rather than forcing point estimates, Shape-Bayes dynamically weights visual evidence against geometric priors to infer a structurally valid shape posterior. Demonstrated on human face shape regression, a rigorous testbed featuring complex non-rigid deformations and strict anatomical constraints, Shape-Bayes comprises: (1) a base model predicting noisy landmarks alongside distilled aleatoric uncertainties; (2) a lightweight Transformer encoding these observations into an adaptive prior over a PCA shape manifold; and (3) a differentiable Bayesian solver computing closed-form posteriors by balancing the noisy predictions against this prior. By guaranteeing complete structural integrity, Shape-Bayes achieves an absolute improvement of up to ~34% IDR over state-of-the-art deterministic models. Simultaneously, it yields highly calibrated uncertainty bounds and reduces relative error by up to 12.5%, establishing a new state-of-the-art for robust 2D face shape regression under severe occlusion. The project page is at https://shape-bayes.github.io.

1 INTRODUCTION

Shape inference is ambiguous under occlusion, yet modern regression often predicts a single shape and can collapse structurally when evidence is missing. SHAPE-BAYES combines uncertainty-aware observations with geometric priors to infer coherent shape posteriors, and its face-regression evaluation reports improved accuracy, structural integrity, and calibration.

Problem

  • Deterministic regressors can produce invalid shapes or generic averages under severe occlusion, while a principled framework for valid shape distributions remains missing.Existing uncertainty-aware facial alignment work still does not address structurally valid distributions under severe occlusion.Source: §1 · p. 2

Approach

  • SHAPE-BAYES uses distilled aleatoric uncertainty to weight local visual evidence against a data-driven geometric prior and infer a coherent shape posterior.Uncertainty acts as a spatial gate indicating which coordinates are reliable and which should rely on anatomical priors.Source: §1 · p. 2, Figure 2 · p. 2

Results

  • Across six base models, SHAPE-BAYES reduces NMEocc by 12.4% for HR-noSA and FR10 by 13.7% for PiPNet, while raising HR-noSA IDR from 66.6% to 100%.For HR-noSA, ECE falls from 7.56 to 1.97.Source: §1 · p. 2

Contributions

  • The framework combines a shape posterior, an adaptive data-driven prior, and a closed-form Bayesian update to balance point-wise evidence against structure.The schematic describes the method as topology-agnostic, although the reported demonstration is face shape regression.Source: Figure 2 · p. 2, Figure 3 · p. 3

2 SHAPE-BAYES

SHAPE-BAYES combines uncertainty-aware regression, a PCA shape manifold, an adaptive prior, and a differentiable Bayesian solver. Its training separates localization accuracy from uncertainty calibration.

2.1 UNCERTAINTY-AWARE SHAPE REGRESSION USING A BASE MODEL

  • Affine-augmented teacher predictions are inverse-transformed, and their empirical spread supervises landmark-wise aleatoric uncertainty without manual annotations.The resulting uncertainty identifies regions where the Bayesian solver should discount visual evidence.Source: Figure 4 · p. 3, §2.1 · pp. 3–4

2 SHAPE-BAYES

  • SHAPE-BAYES combines uncertainty-aware base regression, a Transformer-encoded adaptive shape prior, and exact differentiable Bayesian inference to produce structurally valid posterior shapes.The prior constrains inference to a low-dimensional PCA manifold in the face-alignment testbed.Source: §2 · p. 3, §2.2 · p. 4

2.2 OBSERVED SHAPE LIKELIHOOD AND CONDITIONAL SHAPE PRIOR ESTIMATION

  • The likelihood weights each predicted coordinate by its uncertainty, while a Transformer maps landmark coordinates and uncertainties to an adaptive prior precision in PCA space.Self-attention aggregates global context so confident visible regions can inform heavily occluded ones.Source: §2.2 · p. 4

2.3 DIFFERENTIABLE BAYESIAN SHAPE SOLVER

  • The exact Gaussian posterior adds visual-observation information to prior precision; as measurement precision vanishes for an occluded region, the shape relies on the structural prior.The low-dimensional manifold makes posterior covariance inversion computationally trivial and fully differentiable.Source: §2.3 · pp. 4–5

2.4 TRAINING AND OPTIMIZATION

  • Training combines posterior-mean MSE with posterior NLL, using a stop-gradient on the mean during NLL computation to decouple coordinate accuracy from uncertainty quality.The MSE term drives localization; the NLL term optimizes the predicted posterior precision.Source: §2.4 · p. 5

3 EXPERIMENTAL SETUP

The evaluation tests face-shape inference under dynamic occlusion across three datasets and six diverse base architectures. It measures localization, structural integrity, and posterior calibration.

Datasets and Dynamic Occlusion

  • Evaluation uses test sets from 300W, COFW, and WFLW, with synthetic masks and 80+ real-world objects overlaid to test out-of-distribution occlusion robustness.The protocol preserves ground-truth shapes and identifies occluded landmark indices for evaluation.Source: §3 · p. 5

Base Models and Implementation

  • SHAPE-BAYES is integrated with six base models spanning heatmap, coordinate, and Transformer-based paradigms, using a rank-K PCA manifold and a lightweight five-layer Prior Transformer.The PCA subspace captures more than 99% of structural variance, and the method adds less than 0.2% FLOPs relative to HR-noSA.Source: §3 · pp. 5–6

Evaluation Metrics

  • The evaluation reports global and occluded NME, minNME over 100 posterior samples, FR10, AUC10, IDR, NLL, ECE, and Spearman correlation between predicted variance and true error.IDR measures the percentage of predictions within the learned PCA manifold's 95% confidence hyperellipsoid.Source: §3 · p. 6

4 RESULTS AND DISCUSSION

On severe occlusions, SHAPE-BAYES preserves structural validity while improving face-shape accuracy and posterior calibration. Results span six base models and three test sets, with an ablation showing gains from variance weighting and an adaptive prior.

Structural integrity

  • SHAPE-BAYES achieves 100% IDR across six base models and three test sets, while HR-noSA falls to 66.62% IDR on occluded 300W.COFW’s sparse 29-point topology can leave baseline IDR deceptively high; the PCA-conditioned predictions retain 100% IDR regardless of landmark density.Source: §4 · p. 6, Figure 6 · p. 7

Shape prediction accuracy

  • On 300W, SHAPE-BAYES reduces HR-noSA’s relative NMEocc by 12.4% and relative NMEall by 6.5%, with 100-sample draws extending error reduction to 13%.The posterior mean preserves fine-grained precision on visible landmarks; high-confidence points anchor shape pose while uncertain occluded points are down-weighted.Source: §4 · p. 7, Table 1 · p. 7

Cross-architecture generalization

  • On occluded 300W, PiPNet’s relative NMEocc drops 21.2%, while STAR’s drops 17.8% and its IDR rises from 63.0% to 100%.LUVLi and DSLPT provide native uncertainty without distillation; directly using LUVLi’s uncertainty lowers relative NMEocc by 10.2% on 300W and 7.7% on WFLW.Source: §4 · p. 7, Table 3 · p. 7

Posterior calibration

  • Analytic posterior updates reduce ECE across architectures; HR-noSA reaches 1.97 and LUVLi 0.86, with consistent NLL improvements on occluded 300W.Estimated aleatoric uncertainty correlates with structural errors (Spearman ρ = 0.67), and sparsification tracks the oracle.Source: §4 · p. 8, Table 6 · p. 8, Figure 8 · p. 9

Ablation study

  • On occluded 300W, adding inverse-variance weights lowers NME from 8.85 to 8.08 and FR from 31.57% to 25.69%; posterior sampling yields minNME@100 of 6.79.Unweighted least-squares assumes isotropic noise, so corrupted landmarks can distort the manifold and pull confident landmarks out of place.Source: Figure 7 · p. 8, §4 · p. 9, Table 7 · p. 9
  • The complete dynamic-prior framework achieves NME 7.91, minNME@100 6.60, FR 24.67%, and AUC 31.53% on occluded 300W.The adaptive prior accommodates input-specific variation such as head pose, improving on a potentially rigid static prior.Source: §4 · p. 9, Table 7 · p. 9

5 RELATED WORK

SHAPE-BAYES is positioned against earlier shape-prior, uncertainty-modeling, and occlusion approaches. It combines a Transformer-encoded prior with shape uncertainty in a single-pass Gaussian update.

Probabilistic Shape Inference

  • Classical linear Gaussian subspaces enforce geometric consistency, while generative priors can require expensive sampling and struggle to ingest localized uncertainty analytically.PPL uses categorical priors from discrete memory banks and requires iterative refinement.Source: §5 · p. 9

Uncertainty Modeling in Shape Inference

  • SHAPE-BAYES actively exploits uncertainty by distilling affine-augmented teacher uncertainty into base regressors, then injecting it into a closed-form Bayesian update that arbitrates between noisy local predictions and global priors.Source: §5 · p. 10

Shape Inference under Occlusion

  • Unlike occlusion approaches that lack tightly coupled global shape constraints, SHAPE-BAYES analytically fuses heteroscedastic observations with a conditional shape prior to guarantee structural validity.Compared approaches include explicit occlusion modeling [Kumar et al., 2020], autoregressive joint prediction [Zheng et al., 2025], spatial prior propagation [Zheng et al., 2026], and relational reasoning [Zhu et al., 2022a; Xiang et al., 2026; Chiang et al., 2025].Source: §5 · p. 10

6 CONCLUSION

SHAPE-BAYES unifies distilled aleatoric uncertainty with a learned conditional prior to compute exact closed-form posteriors. Across six face-alignment architectures, it reduces shape error, sharpens uncertainty calibration, and enforces structural validity.

6 CONCLUSION

  • SHAPE-BAYES computes exact closed-form posteriors by unifying distilled aleatoric uncertainty with a learned conditional prior.This balances noisy local evidence against global structural constraints and prevents structural collapse under severe occlusion.Source: §6 · p. 10
  • Across six face-alignment base architectures, SHAPE-BAYES consistently reduces shape error and sharpens uncertainty calibration.Source: §6 · p. 10
  • SHAPE-BAYES enforces structural validity, providing a principled solution for structured shape inference.Source: §6 · p. 10

A QUALITATIVE RESULTS ON OCCLUDED IMAGE TEST SETS

Qualitative comparisons on dynamically occluded test images show that SHAPE-BAYES corrects structural failures from HR-noSA across 300W, COFW, and WFLW. Its Bayesian update projects noisy observations onto a learned shape manifold to recover plausible facial geometries.

A QUALITATIVE RESULTS ON OCCLUDED IMAGE TEST SETS

B REGION-WISE SHAPE ACCURACY IMPROVEMENTS ANALYSIS

SHAPE-BAYES reduces occluded landmark error across facial regions, datasets, and base models. The accompanying computational analysis describes the Transformer configuration and parameter distribution for the 300W setup.

B REGION-WISE SHAPE ACCURACY IMPROVEMENTS ANALYSIS

C SHAPE-BAYES TRANSFORMER: COMPUTATIONAL COMPLEXITY ANALYSIS

  • The computational-cost analysis benchmarks the Transformer encoder and closed-form Bayesian solve against HR-noSA’s StackedHGNetV1 network.Its figures use the 300W configuration with N = 68 landmarks and K = 60 PCA components, with counts verified against PyTorch parameter counts.Source: C SHAPE-BAYES TRANSFORMER:… · p. 17

C.1 SHAPE-BAYES TRANSFORMER ARCHITECTURE

  • The Transformer maps each landmark’s observation pair (s_i, σ_i) in R^4 to a K-dimensional log-precision vector.The encoder uses d = 64, dff = 128, L = 5, and H = 8 attention heads.Source: §C.1 · p. 17

C.2 PARAMETER COUNT

  • The precision head accounts for 60.9% of parameters, while the five Transformer layers contribute 167,360 parameters (39.0%).The passage attributes the head’s share to flattening all N token embeddings before the final linear projection.Source: §C.2 · p. 17, Table 11 · p. 17

C.3 FLOPS ANALYSIS

The analysis finds the SHAPE-BAYES module adds little computation beside the base network, with the Transformer as its main cost. Its forward pass adds under 0.5 ms per image.

C.3 FLOPS ANALYSIS

  • The MAP solve uses Cholesky factorization and two triangular solves, while posterior sampling additionally requires a matrix inverse.The (P⊤W)P product costs 979K FLOPs, ∼90% of the solve.Source: §C.3 · p. 18

C.4 COMPARISON WITH THE BASE LANDMARK REGRESSION NETWORK

  • At ∼29.8–30.3M FLOPs, the SHAPE-BAYES module is approximately 0.10% of the base model’s ∼29.1 GFLOPs; the Transformer accounts for 94.6% of module FLOPs.The closed-form solve contributes 1.1–1.5M FLOPs (∼3.6–5.1%).Source: §C.4 · p. 19
  • The forward pass adds < 0.5 ms per image on an NVIDIA RTX 3090, compared with ∼12 ms for the base model’s convolutional backbone.Posterior sampling adds < 0.1 ms per sample.Source: §C.4 · p. 19

D SHAPE-BAYES TRAINING AND OPTIMIZATION

The SHAPE-BAYES encoder maps predicted coordinates and uncertainties to a diagonal prior precision matrix. It is trained offline with dataset-specific PCA dimensions and a joint shape-accuracy and uncertainty objective.

D SHAPE-BAYES TRAINING AND OPTIMIZATION

E BASE MODEL TRAINING AND IMPLEMENTATION DETAILS

The base-model training pipeline distills landmark-wise uncertainty from augmented predictions of frozen teachers. It adapts uncertainty supervision to coordinate- and heatmap-regression models, while using native uncertainty outputs where available.

E BASE MODEL TRAINING AND IMPLEMENTATION DETAILS

  • A frozen teacher produces 30 augmented views per image, which are mapped back to the original frame to estimate per-landmark uncertainty alongside the point prediction.The views vary spatial transformations, photometric conditions, and horizontal flipping.Source: E BASE MODEL TRAINING AND… · p. 20
  • Distilled σ′ is clipped to [10−6, 0.5 × dIOD], and models with native uncertainty estimates are used without fine-tuning.LUVLi and DSLPT provide native uncertainty; four other baselines receive uncertainty prediction capabilities.Source: E BASE MODEL TRAINING AND… · p. 20
  • Coordinate-regression models gain a positive variance head trained with Gaussian NLL, while heatmap models use anisotropic Gaussian targets derived from distilled uncertainty.The predicted heatmap spread then encodes aleatoric uncertainty for inference.Source: E BASE MODEL TRAINING AND… · pp. 20–21

F EVALUATION METRIC DEFINITIONS

The appendices define evaluation measures and detail how occlusions and prior-training variants are synthesized. They also describe a sampling scheme intended to cover posterior modes for minNME evaluation.

F EVALUATION METRIC DEFINITIONS

  • NME is evaluated over all or occluded landmarks, IDR requires both PCA-space confidence-region membership and a bounded reconstruction error, and predictive uncertainty combines epistemic and aleatoric components.Source: F EVALUATION METRIC DEFINITIONS · p. 21

G DYNAMIC OCCLUSION: ASSET CATALOGUE AND PLACEMENT SPECIFICATIONS

G.1 PIPELINE 1: WEARABLE FACE MASKS (MASKTHEFACE)

  • Wearable masks span five types and are fitted to facial geometry with a six-point affine warp before compositing and per-landmark occlusion labeling.Source: §G.1 · p. 21

G.2 PIPELINE 2: REAL-WORLD OBJECT OVERLAYS

  • Object overlays are scaled to a sampled face-area percentage using opaque pixel area, and landmarks are labeled occluded when overlay alpha at their location exceeds 0.5.Source: §G.2 · p. 22

H DATA-DRIVEN PRIOR: TRAINING AUGMENTATION PROTOCOL

  • The prior encoder is trained on 13 procedural variants, with a 4:9 ratio of non-occluding to occluding variants so it also sees confident predictions.This ratio is intended to prevent the prior from becoming unconditionally strong.Source: H DATA-DRIVEN PRIOR: TRAINING… · p. 22

H.1 NON-OCCLUDING PHOTOMETRIC VARIANTS

  • Non-occluding photometric variants degrade image quality without directly covering landmarks, leaving the base model largely confident.Examples include brightness, contrast, and color shifts.Source: §H.1 · p. 22

H.2 OCCLUDING STRUCTURAL VARIANTS

  • Occluding training variants include eye and lower-face blocks, random cutouts, and sensor noise; frozen-base predictions over the variants are collected with 30 TTA passes.Source: §H.2 · p. 23

I DIVERSE POSTERIOR SAMPLING FOR MINNME

  • For minNME, the first 2K posterior samples lie at the 95th-percentile boundary along each independent PCA axis to cover principal modes.Additional samples, when needed, use standard multivariate-normal sampling.Source: I DIVERSE POSTERIOR SAMPLING FOR MINNME · pp. 23–24

Related papers

Loading 2610.09032v1…