STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection

1. Introduction

• Facial landmark detection locates predefined facial landmarks and serves downstream tasks including face verification, face synthesis, and 3D face reconstruction.The paper describes coordinate regression and heatmap regression with differentiable soft-Argmax as the setting in which regression loss fits predictions to manual annotations.Source: §1 · p. 1
• Ambiguous contour landmarks lack clear positions, causing inconsistent annotations that can mislead regression loss and harm convergence and performance.The paper identifies mitigating the impact of semantic ambiguity as the central regression-loss design problem.Source: §1 · pp. 1–2
• The authors infer that semantic ambiguity is associated with anisotropic predicted landmark distributions, especially for facial contour points.They contrast contour-point distributions with the isotropic distribution of an eye-corner point.Source: §1 · p. 2
• The first principal component aligns with the ambiguity direction for contour landmarks, based on visualized distributions and components.The paper introduces customized PCA for discrete probability distributions, using weighted mean and covariance estimation followed by eigen-decomposition.Source: §1 · p. 2
• STAR loss decomposes prediction error along two principal-component directions and scales by their energy, suppressing error in the higher-energy direction; eigenvalue restrictions address abnormal loss decrease.The authors report that unrestricted energy increase can lead to premature convergence.Source: §1 · p. 2
• STAR loss achieves competitive performance against state-of-the-art methods on COFW, 300W, and WFLW.The evaluation covers all three benchmarks.Source: §1 · p. 2

2. Related Work

• Deep facial landmark methods mainly use coordinate or heatmap regression, with CNN-based approaches achieving state-of-the-art performance.Earlier statistical approaches include Active Appearance Models, Constrained Local Models, and 3D Morphable Models.Source: §2 · pp. 2–3
• Coordinate-regression work addresses training-sample demands through data methods, cascaded refinement, graph constraints, and attention-based landmark relations.Heatmap methods use architectures such as stacked hourglass, U-Net, and HRNet, while quantization error motivates improved heatmaps, decoding, and coordinate refinement.Source: §2 · p. 3
• Prior ambiguity approaches include optical-flow supervision, facial-boundary constraints, and latent-variable annotation optimization, each with stated sensitivity, cost, or training-complexity concerns.SBR is sensitive to illumination and occlusion; LAB is computationally expensive; SA uses a complex training strategy.Source: §2 · p. 3
• ADNet's Anisotropic Direction Loss constrains facial-boundary landmarks more strongly in the normal than tangent direction, using hand-crafted direction and constraint weights.The paper characterizes that design as coarse-grained and performance-degrading.Source: §2 · p. 3

3. Preliminary

• Four stacked Hourglass networks generate landmark heatmaps, which are treated as probability distributions and decoded into coordinates by soft-Argmax.Each Hourglass generates one heatmap per predefined facial landmark.Source: §3 · p. 3
• The regression loss measures a distance between predicted coordinates and manual annotations, with L1, L2, smooth-L1, and Wing distances among common choices.Predicted coordinates are decoded from the heatmap's probability distribution.Source: §3 · p. 3
• The paper locates the semantic-ambiguity problem in regression loss applied to ambiguous manual annotations.Source: §3 · p. 3

4.1. Anaysis of the Semantic Ambiguity

• The customized PCA approximates the mean and covariance of a discrete probability distribution, then computes the covariance matrix’s eigen-decomposition.The PCA visualization encodes the first and second principal components as arrows, with lengths corresponding to eigenvalues and ellipse shading representing λ1/λ2.Source: §4.1 · p. 4, Figure 3 · p. 4
• For contour landmarks, the first principal component aligns with the face contour and with the ambiguity direction; greater eccentricity accompanies more severe ambiguity.The authors infer that principal-component energy can represent ambiguity intensity because it indicates directional variance.Source: §4.1 · p. 4

4.2. STAR Loss

• STAR loss decomposes prediction error along two principal component directions and divides each component by its corresponding energy to reduce ambiguity effects adaptively.The projected errors use the selected distance function, which the authors state can be arbitrary.Source: §4.2 · p. 4
• When λ1 greatly exceeds λ2, STAR loss reduces the model’s attention to error along the first principal component, which is mainly attributed to inconsistent manual annotation.The authors state this can mitigate semantic ambiguity’s impact and lead to better convergence.Source: §4.2 · p. 4
• The authors propose value and detach restrictions to prevent abnormal eigenvalue increases, and report that both help mitigate the problem.Value restriction directly constrains eigenvalue increases; detach restriction cuts off gradients through eigenvalues and eigenvectors, treating them as constants in STAR loss.Source: §4.2 · p. 5

5.1. Experiment Settings

• Experiments use a four-stacked hourglass backbone and evaluate against 14 baselines on COFW, 300W, WFLW, and cross-dataset 300VW.Training uses 256 × 256 inputs; evaluation reports NME, FR, and AUC.Source: §5.1 · p. 5, §5.2 · p. 5

5.2. Accuracy Evaluation

• STAR achieves state-of-the-art results on 300W test sets and COFW, the best WFLW results, and a 0.2 average improvement on 300VW Category-C.On WFLW, the reported gains over state-of-the-art methods are 0.06 NME and 0.28 FR10%.Source: Table 1 · p. 6, §5.2 · p. 6, Table 2 · p. 6

5.3. Ablation Study

• STAR improves the regression-loss baseline by averages of 0.34 on COFW, 0.19 on 300W, and 0.25 on WFLW across distance functions.The effect is significant when STAR is combined with l2-distance.Source: §5.3 · p. 7
• STAR's qualitative WFLW results show better structural constraints, and the authors infer it acts as label regularization against overfitting ambiguous annotations.Figure 5 marks clear failures solved with STAR.Source: §5.3 · p. 7, Figure 5 · p. 7
• DR and AAM each improve STAR across WFLW (0.07, 0.16), COFW (0.14, 0.35), and 300W (0.11, 0.09); AAM is better suited, and smooth-l1+AAM+STAR performs best.AAM's boundary information is given as the reason it pairs better with STAR.Source: Table 3 · p. 6, §5.3 · p. 7
• STAR improves results at 64px, 128px, and 256px by averages of 0.20, 0.30, and 0.25, and its 128px baseline matches or beats the 256px baseline.The value restriction is not sensitive to w, for which the default is 1; detach restriction is essential and comparable to value restriction.Source: §5.3 · pp. 7–8, Table 4 · p. 8
• Across five same-setting models, mean prediction variance falls from 0.79 to 0.69 with STAR, whose predictions cluster more tightly.The authors interpret this as reduced impact of ambiguous annotations and more stable predictions.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4. Discussion with Related Solutions

• Against ADL, STAR estimates direction and constraint weight through heatmap PCA; against Mahalanobis distance, STAR records 4.17 versus 4.32 NME on WFLW.The authors describe STAR as plug-and-play for heatmap methods and lighter than LUVLi, without extra direction-computation design.Source: §5.4 · p. 8, Table 6 · p. 8

6. Conclusion

• STAR uses PCA of predicted distributions to estimate the direction and intensity of semantic ambiguity.The conclusion says predicted distributions can represent semantic ambiguity.Source: §6 · p. 8
• STAR suppresses prediction error in the ambiguity direction and achieves state-of-the-art performance on three benchmarks.Source: §6 · p. 8

A.1. Discussion on the Motivation of STAR

• STAR adapts to ambiguity that varies across samples and landmarks, unlike ADNet’s constant hand-crafted constraint weight.The discussion says a higher λ1/λ2 indicates more severe ambiguity.Source: §A.1 · p. 11

A.2. Discussion on the Influence of Semantic Ambiguity on STAR

• The discussion attributes COFW’s less obvious improvement relative to WFLW to COFW landmarks being on facial features with relatively precise semantics.Source: §A.2 · p. 11
• STAR is reported to improve performance even with relatively precise semantics, though its gains are more obvious when ambiguity is severe.Source: §A.2 · p. 11
• 1.06 versus 0.52 variance scores and 0.23 versus 0.12 NME improvements indicate larger STAR gains on the more ambiguous face contour.The table caption describes the comparison as STAR’s more obvious improvement in datasets with severe ambiguity.Source: Table 8 · p. 11, §A.2 · p. 11

A.3. Discussion on the Effect of STAR on Hard Samples

• STAR shows significant improvement on challenge test sets, where most hard samples occur.Source: §A.3 · p. 11
• The discussion says hard samples have more serious ambiguity than easy samples, leaving more room for improvement.Source: §A.3 · p. 11
• STAR acts as label regularization that emphasizes structural constraints between landmarks, which the discussion says significantly affects hard-sample detection.Source: §A.3 · p. 11

B. Full Experiments

• The full experiments report WFLW subset NME, FR0.1, and AUC0.1, COFW NME and FR0.1, and 300W NME.Source: B. Full Experiments · p. 11

B.1. Details of Comparison on COFW

• COFW comparisons cover inter-ocular and inter-pupil normalization, with Table 9 presenting NME and FR0.1.Source: §B.1 · p. 11, Table 9 · p. 12

B.2. Details of Comparison on 300W

• The 300W comparisons include inter-pupil and inter-ocular normalization results in Tables 10 and 11.Source: Table 10 · p. 12, Table 11 · p. 12, §B.2 · p. 12

B.3. Details of Comparison on WFLW

• STAR achieves competitive state-of-the-art-level NME, FR0.1, and AUC0.1 performance on the WFLW test set and all its subsets.Source: §B.3 · p. 12

• Table 12 compares STAR with state-of-the-art methods on WFLW and its subsets, marking the best and second-best results in red and blue.Source: Table 12 · p. 13
• Figure 7 shows WFLW principal components estimated from heatmaps via PCA, indicated by yellow and blue arrows.Source: §C.1 · p. 12, Figure 7 · p. 14
• The blue ellipse’s shading represents ambiguity strength.Source: Figure 7 · p. 14