PaperlayerView exact revision on arXiv

Source-linked AI summary

FreqFLD: Towards All-in-One Facial Landmark Detection via Frequency Modulation

Shun Ren, Kaijie Jin, Shengkai Hu, Beihang Song, Hang Sun, Wenwen Min, Youfa Liu, Jun Wan

arXiv:2609.10278v1cs.CV

TL;DR

Existing facial landmark detection methods are often optimized for individual datasets, limiting unified cross-dataset modeling. FreqFLD introduces a frequency-modulated All-in-One framework integrating frequency-aware components, achieving comparable performance across multiple face alignment benchmarks while leaving computational overhead for future reduction.

  • Problem

    Existing facial landmark detection methods are independently optimized for individual benchmarks, while All-in-One learning seeks a unified model that generalizes across datasets.

  • Method

    FreqFLD is a frequency-modulated All-in-One framework that jointly integrates FreqMoM, FreqMoE, and FreqCR for facial landmark detection.

  • Results

    FreqFLD achieves comparable performance across multiple face alignment benchmarks.

  • Takeaways & Limitations

    The integrated framework provides coherent frequency-aware modeling for All-in-One facial landmark detection across multiple benchmarks.

  • Takeaways & Limitations

    The authors identify reducing computational overhead while preserving robustness as future work.

Abstract

from arXiv · show

Recent progress in deep learning has significantly advanced facial landmark detection. However, most existing methods process features in a spatial-domain manner under a dataset-specific training paradigm, which overlooks the fact that facial landmark detection is inherently geometry-driven and sensitive to frequency variations, thereby limiting cross-dataset generalization under complex scenarios and hindering the development of a facial landmark detection model. To address this issue, we propose \textbf{FreqFLD}, a \textbf{freq}uency-modulated framework towards All-in-One \textbf{f}acial \textbf{l}andmark \textbf{d}etection. Specifically, FreqFLD introduces a Frequency Modulation Module (FreqMoM) to explicitly induce the frequency prior by decoupling and modulating low- and high-frequency components, which is then injected into subsequent feature modeling to enable balanced modeling of global facial structure and local landmark details. Furthermore, FreqFLD employs a Frequency-Modulated Mixture-of-Experts (FreqMoE), with expert selection adaptively conditioned on frequency-modulated priors, enabling flexible modeling of heterogeneous facial landmark patterns under diverse and challenging scenarios. To regularize frequency-consistent modeling under the All-in-One paradigm, we further introduce a Frequency-Consistent Routing (FreqCR) loss, which constrains the routing and assignment of frequency-aware experts to promote balanced expert utilization across diverse facial scenarios, thereby enabling stable expert specialization and achieving robust facial landmark detection. Extensive experiments demonstrate that the proposed FreqFLD achieves comparable performance on popular datasets. The code is available at: https://github.com/jkj1059657014/FreqFLD.

1. INTRODUCTION

Facial landmark detection benefits from a unified model because heterogeneous datasets share fundamental facial geometry, yet dataset-specific and spatial-domain approaches struggle with cross-dataset variation. FreqFLD addresses this through frequency-aware feature modulation, adaptive expert routing, and routing regularization, achieving comparable performance across multiple benchmarks.

  • 1. INTRODUCTION: Existing dataset-specific FLD methods require repeated training and show limited generalization across datasets despite shared facial geometry.The benchmarks use different landmark annotation schemes, while overlapping semantic landmarks span a shared geometric subspace.
  • 1. INTRODUCTION: FreqFLD combines a Frequency Modulation Module, Frequency-Modulated Mixture-of-Experts, and Frequency-Consistent Routing loss for All-in-One facial landmark detection.These components jointly provide frequency-aware representation learning and stable expert specialization.
  • 1. INTRODUCTION: FreqMoM decouples low- and high-frequency features to model global facial structure and landmark-sensitive details separately.The resulting frequency-disentangled priors are injected into subsequent feature modeling.
  • 1. INTRODUCTION: FreqMoE conditions expert routing on frequency-aware structural cues to capture heterogeneous facial patterns while preserving shared geometric structure.This design targets feature conflicts caused by diverse facial conditions and datasets.
  • 1. INTRODUCTION: FreqCR constrains expert assignment to frequency-aware cues, stabilizing routing and preventing expert imbalance during All-in-One training.The stated effect is more stable optimization and improved cross-dataset generalization.
  • 1. INTRODUCTION: FreqFLD achieves comparable facial landmark detection performance across multiple benchmarks while alleviating gradient conflicts in multi-dataset training.The framework is designed for unified learning across heterogeneous facial landmark datasets.

2. RELATED WORK

Existing facial landmark detection methods include handcrafted, coordinate-regression, and heatmap-regression approaches, but predominantly rely on spatial-domain modeling and dataset-specific supervision. These limitations motivate a unified framework for robust cross-dataset detection under diverse facial conditions.

  • 2.1. Coordinate Regression Methods: Coordinate regression directly predicts landmark locations but remains sensitive to facial geometry and spatial configurations under large poses, occlusions, and complex expressions.
  • Prior work improves facial landmark modeling through structural dependencies, including coarse-to-fine frameworks, self-attention, graph relations, and sparse adaptive interactions.
  • 2.2. Heatmap Regression Methods: Heatmap regression provides dense spatial supervision and strong robustness, but remains sensitive to domain shifts and limited cross-dataset generalization.
  • Existing FLD methods mainly rely on dataset-specific optimization and spatial-domain modeling, limiting generalization across diverse datasets and scenarios.
  • The proposed FreqFLD addresses these limitations with FreqMoM, FreqMoE, and adaptive expert routing regularized by FreqCR for All-in-One FLD.

3. Method

The method section introduces a unified landmark index, the overall FreqFLD pipeline, its frequency-modulated modules, and the FreqCR training loss.

  • The method is organized around a unified landmark index, an overall pipeline, FreqMoM, FreqMoE, and FreqCR loss.

3.1. Unified Landmark Index

FreqFLD establishes a universal landmark representation across multiple facial landmark datasets, using shared semantics and indexing to support unified detection. The framework then models frequency-aware features and predicts the unified landmark index.

  • The multi-scale encoder-decoder progressively refines features before a prediction head outputs the unified landmark index.
  • FreqMoM decomposes bottleneck features into low- and high-frequency components, refines them into frequency-aware prompts, and uses those prompts to guide FreqMoE expert activation.
  • 124 universal landmarks with defined semantics and unified indexing merge annotations from 300W, WFLW, COFW, and AFLW.
  • Shared landmark annotations across diverse datasets are intended to improve overall detection accuracy.

3.2. Overall Pipeline

FreqFLD combines hierarchical encoder-decoder features with frequency modulation and adaptive mixture-of-experts routing. FreqCR regularizes expert assignment during All-in-One training before unified landmark prediction.

  • FreqFLD is a frequency-modulated All-in-One framework that models heterogeneous facial patterns.
  • The pipeline preprocesses facial images, extracts hierarchical features with a multi-stage encoder-decoder and skip connections, and fuses them progressively.
  • FreqMoM decomposes features into complementary low-frequency structural and high-frequency detail components, producing priors for subsequent modeling.
  • Frequency-aware representations condition adaptive FreqMoE expert routing to support flexible modeling of heterogeneous facial patterns.
  • FreqCR regularizes expert assignment during All-in-One training before the final prediction stage produces unified landmark outputs.

3.3. Frequency Modulation Module (FreqMoM)

FreqMoM explicitly decouples low- and high-frequency facial features, then modulates both using frequency-specific prompts to model global geometry and local landmark details.

  • Unified Frequency Decoupling Module: The Unified Frequency Decoupling Module learns an adaptive low-pass filter and obtains high-frequency features by subtracting low-frequency features from the input.
  • FreqMoM separates low-frequency components for global facial geometry from high-frequency components for fine-grained landmark details.
  • Unified Frequency Modulation Module: The Unified Frequency Modulation Module feeds the two frequency priors into separate High-Frequency and Low-Frequency Prompt Blocks before concatenating their refined features.
  • High-frequency Prompt Block (HFPB): The High-Frequency Prompt Block uses depthwise convolution, gating, a learnable prompt, and cross-attention to emphasize geometrically informative local details.
  • Low-frequency Prompt Block (LFPB): The Low-Frequency Prompt Block applies FFT-domain gating and a learnable prompt, then cross-attention, to produce coherent global facial structure.

3.4. Frequency Mixture-of-Experts (FreqMoE)

FreqMoE combines shared structural modeling with frequency-adaptive experts, using image-level frequency-conditioned routing and scale-varying receptive fields for heterogeneous facial patterns.

  • FreqMoE addresses heterogeneous pose and structural complexity through a shared structural path and a frequency-adaptive expert path.
  • Shared Structural Expert Modeling: The shared structural expert uses MHSA to aggregate long-range facial dependencies and encode global geometry as a common reference.
  • Frequency-aware Image-level Expert Routing: Routing each image as a whole to one expert preserves global facial structure and scale consistency while avoiding fragmented token-level assignments.
  • Frequency-aware Image-level Expert Routing: Image-level routing combines logits from the input representation and refined frequency-aware features, then applies noisy Softmax Top-k selection.
  • Expert Design: Frequency-adaptive experts share a frequency-modulated architecture but use progressively larger patches, balancing local landmark detail with broader facial geometry.

3.5. Frequency-Consistent Routing (FreqCR) Loss

FreqCR regularizes frequency-aware expert routing under All-in-One training by balancing utilization while accounting for heterogeneous expert capacities and noisy assignment behavior.

  • FreqCR explicitly regularizes expert utilization to remain balanced across experts and consistent with their computational capacities.
  • The loss combines complexity-aware importance and load-balancing terms to address imbalanced utilization, routing instability, and expert collapse.
  • Complexity-aware importance loss: A fixed parameter-count prior assigns each expert a normalized complexity bias for complexity-aware importance regularization.
  • Load balancing loss: The load-balancing term estimates expected expert assignment under noisy Top-k routing using Gaussian perturbations and selection probabilities.
  • The overall objective integrates the facial-landmark MSE loss with the routing regularizers weighted by λ1 and λ2.

4. EXPERIMENTS

FreqFLD is evaluated against state-of-the-art facial landmark detection methods across four benchmarks using normalized landmark errors and failure rate. It achieves competitive or superior performance across standard and challenging subsets, while ablations and analyses examine its frequency modulation, expert routing, and cross-dataset generalization.

  • 4.2. Comparison with State-of-the-Art Methods: 4.80 NMEip and 0.39 failure rate on COFW outperform most recent facial landmark detection methods, while 5.53 NMEio on WFLW improves over existing approaches.The paper also reports stable accuracy and competitive robustness on the 300W challenging subset.
  • 4.2. Comparison with State-of-the-Art Methods: 4.52 NMEio on the 300W challenging subset outperforms several state-of-the-art methods, while AFLW remains comparable and WFLW pose and expression subsets improve over existing approaches.The reported results also indicate robustness to extreme facial variations.
  • 4.2. Comparison with State-of-the-Art Methods: 4.55 and 5.15 NMEio on WFLW illumination and blur subsets surpass several representative methods, maintaining accuracy under poor image quality.The cited results specifically address illumination and severe blur degradation.
  • 4.3. Ablation Studies: 4.52 NMEio is achieved by the full model with FreqCR, improving by 0.19 over the baseline and by 0.08 and 0.10 over variants using FreqMoE or FreqMoM alone.The ablation attributes complementary gains to frequency modulation and mixture-of-experts modeling.
  • 4.4. Self-Evaluation Analysis: 4.74 NMEio on 300W, 4.85 on WFLW, 5.13 NMEip on COFW, and 1.88 NMEbox on AFLW are reported for held-out datasets.These results are presented as evidence of transfer to unseen annotation distributions and facial appearance statistics.
  • 4.4. Self-Evaluation Analysis: FreqMoM produces compact intra-dataset clusters and clearer inter-dataset separation, while FreqMoE routes simpler datasets to lightweight experts and more complex datasets to higher-capacity experts.The analyses associate these patterns with reduced feature conflicts and adaptive capacity allocation.

5. CONCLUSION

The conclusion presents FreqFLD as an All-in-One facial landmark detection framework that uses frequency modulation, adaptive experts, and routing regularization to address heterogeneous multi-dataset training. It reports comparable performance across benchmarks while identifying computational overhead as a direction for future work.

  • 5. CONCLUSION: FreqMoM disentangles low- and high-frequency facial cues to balance global structure and local details, while FreqMoE routes samples according to facial complexity.FreqMoE assigns samples to experts with appropriate representational capacity for heterogeneous facial data.
  • 5. CONCLUSION: FreqCR stabilizes expert assignment and mitigates expert imbalance, promoting consistent and robust expert specialization across diverse facial scenarios.Its role is described as regularizing routing within the All-in-One setting.
  • 5. CONCLUSION: FreqFLD combines FreqMoM, FreqMoE, and FreqCR to alleviate feature and gradient conflicts in multi-dataset training, achieving comparable performance across face alignment benchmarks.The framework is intended to generalize across variations in pose, expression, and appearance.
  • 5. CONCLUSION: The authors identify lightweight frequency-aware expert designs as future work to reduce computational overhead while preserving robustness on heterogeneous datasets.This is stated as a future research direction rather than a reported limitation of benchmark accuracy.
Loading 2609.10278v1…