LEAF: Unveiling Two Sides of the Same Coin in Semi-supervised Facial Expression Recognition

1. Introduction

• Label scarcity, high inter-class similarity, and potential annotator errors make FER with limited labeled data a practical challenge.The paper motivates using a small labeled set alongside abundant unlabeled data.Source: §1 · pp. 1–2
• Existing semi-supervised FER methods primarily improve pseudo-label quality or quantity, while overlooking expression-relevant representations needed to capture subtle differences.The paper identifies representation quality as the other side of the problem.Source: §1 · p. 2
• LEAF hierarchically decouples expression-agnostic and expression-relevant representations and pseudo-labels, then assigns them different weights through fusion.The framework aims to improve representation quality and pseudo-label quality and quantity.Source: §1 · p. 2
• At semantic and instance levels, learnable gating weights fuse specialized representations; at category level, ambiguous consistency compares predictions from two forward processes.The category-level strategy assigns candidate pseudo-labels as positive or negative, corresponding to expression-relevant or expression-agnostic parts.Source: §1 · p. 2
• LEAF consistently outperforms state-of-the-art approaches across benchmark datasets and label-scarcity settings, while its EAF strategies improve existing methods when integrated.The introduction attributes these findings to extensive experiments on benchmark datasets.Source: §1 · p. 2

2.1. Facial Expression Recognition

• Existing semi-supervised FER approaches focus on pseudo-label quality or quantity while overlooking the potential to improve expression representations.The paper places this gap against data-hungry FER methods and the challenge of label scarcity.Source: §2.1 · p. 3

2.2. Semi-Supervised Learning

• Threshold-based SSL methods select high-confidence unlabeled samples using fixed or dynamic thresholds, but leave some low-confidence samples unused.These approaches can combine weak and strong augmentations for consistency regularization.Source: §2.2 · p. 3
• LEAF applies category-level ambiguous pseudo-label selection so all unlabeled data can contribute to consistency regularization.Source: §2.2 · p. 3

3. Problem Definition

• Semi-supervised FER learns model parameters by combining supervised and unsupervised losses over labeled and unlabeled samples.The unsupervised consistency loss encourages consistent predictions for unlabeled samples under different augmentations.Source: §3 · p. 3
• Existing approaches treat representations equally despite their differing discriminative power, potentially overlooking more informative expression features.Source: §3 · p. 3
• Hard pseudo-labels can be unreliable when labeled data is scarce, limiting the quality of supervision for unlabeled samples.Source: §3 · p. 3

4.1. Semantic-level Decoupling and Fusing

• Semantic-level EAF routes deep features through experts and a learnable gate, using top-K selection to focus on the most relevant experts for each input.The gating network is a feed-forward network; excluded experts receive zero weight after softmax.Source: §4.1 · p. 4

4.2. Instance-level Decoupling and Fusing

• Instance-level EAF applies expert-and-gate fusion to classifier predictions, addressing inter-class similarity that also persists in the low-dimensional category distribution.Its top-K gating adapts expert weights to each instance and is intended to mitigate noisy or ambiguous predictions.Source: §4.2 · p. 5

4.3. Category-level Decoupling and Fusing

• Category-level EAF partitions strong-view predictions into positive and negative label sets, then regularizes weak-view predictions to rank positives above negatives.The positive set is determined from sorted cumulative probabilities using a threshold of 0.9; the margin defaults to zero.Source: §4.3 · p. 5

5.1. Datasets and Evaluation Metrics

• The evaluation covers RAF-DB, FERPlus, and AffectNet variants, datasets described as varying in size and expression granularity.The datasets contain seven or eight expression categories, depending on the dataset or variant.Source: §5.1 · p. 6
• RAF-DB has 12,271 training and 3,068 test images, while FERPlus has 24,941 training, 3,589 validation, and 3,589 test images.RAF-DB has seven expression categories; FERPlus has eight.Source: §5.1 · p. 6
• AffectNet has 286,564 training and 4,000 test images; AffectNet7 excludes contempt, and reported results use balanced accuracy unless specified otherwise.AffectNet7 is described as having 3,667 training and 500 test images for the excluded-expression category.Source: §5.1 · p. 6

5.2. Implementation Details

• Images are aligned and resized to 224 × 224 pixels, and models are trained for 20 epochs with Adam at a learning rate of 5e−4 and batch size 64.Experiments run using PyTorch on NVIDIA Tesla A100 GPUs.Source: §5.2 · p. 6
• LEAF and baselines use ResNet-18 pretrained on MS-Celeb-1M as the backbone, with each baseline's hyperparameters set according to its paper.The authors extend the baselines to the FER domain.Source: §5.2 · p. 6
• The comparison excludes fully supervised FER methods because they require large amounts of labeled data and cannot use unlabeled facial-expression samples to improve recognition.Source: §5.2 · p. 6

5.3. Quantitative Comparison

• Semi-supervised FER methods such as AdaCM, LION, and LEAF outperform semi-supervised image-classification methods in most cases.The authors link this pattern to FER-specific approaches addressing subtle inter-class differences in facial expressions.Source: §5.3 · p. 6
• LEAF consistently outperforms all other approaches across four benchmark datasets and varied settings.The authors attribute this result to semantic- and instance-level representation learning together with category-level assignment of pseudo-labels to ambiguous samples.Source: §5.3 · p. 7, Table 1 · p. 7

5.4. Qualitative Analysis

• As label quantities increase, all methods improve, while LEAF’s performance gap remains significant on RAFDB and FERPlus.The reported comparison spans label quantities from 100 to 500.Source: §5.4 · p. 7
• LEAF maintains a clear advantage over competing methods across datasets and label quantities from 100 to 1000.RAFDB and FERPlus use 100–500 labels; AffectNet7 and AffectNet8 use 500–1000 labels.Source: §5.4 · pp. 7–8
• LEAF’s balanced accuracy shows stable improvements over other methods on AffectNet7 and AffectNet8.The authors identify this pattern as evidence of effectiveness in handling class imbalance.Source: §5.4 · p. 8

5.5.1. Impact of Expert Structure

• Residual experts improve on bottleneck experts, whose nonlinear structure improves performance over linear experts across four datasets.The authors report that residual connections mitigate potential overfitting of similar expressions.Source: §5.5.1 · p. 8

5.5.2. Contribution of Proposed Components

• Removing any EAF level reduces performance, category-level EAF has the largest impact, and the full model performs best across settings.Replacing the ambiguous consistency loss with standard cross-entropy is less suitable for semi-supervised FER.Source: §5.5.2 · p. 8

5.5.3. Sensitivity to the Number of Experts

• The best expert count is 2 on RAFDB and 4 on FERPlus; increasing the count improves performance in most cases, except with 6 experts on RAFDB.With one expert, EAF reduces to a single projection network.Source: §5.5.3 · p. 8, Figure 6 · p. 9

5.5.4. Integration with Other Semi-Supervised Methods

• Integrating EAF into FixMatch and FlexMatch improves performance across label quantities from 100 to 300, with greater gains at 100 labels.The authors report this integration result on RAFDB.Source: §5.5.4 · p. 9, Figure 7 · p. 9

5.6. Visualization Analysis

• LEAF is slightly ahead of AdaCM in compactness and separability of expression features under label scarcity.The visualizations use features from RAFDB and FERPlus with 1600 labeled samples; each expression has a distinct color.Source: §5.6 · p. 9

• LEAF outperforms state-of-the-art methods across varied label-scarcity scenarios, with quantitative and qualitative results supporting its effectiveness and robustness.Source: §6 · p. 9
• LEAF decouples and fuses expression features and predictions to focus on expression-relevant representations and assign expression-relevant pseudo-labels to unlabeled samples.Source: §6 · p. 9
• The authors plan to extend LEAF to dynamic video-based FER and multimodal emotion recognition.Source: §6 · p. 9