PaperlayerView exact revision on arXiv

Source-linked AI summary

POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition

Ce Zheng, Matias Mendieta, Chen Chen

arXiv:2204.04083v2cs.CVcs.AI

TL;DR

FER must handle subtle differences between expressions, visual variation within categories, and changing image scales. POSTER combines image and facial-landmark streams through transformer-based cross-fusion and a feature pyramid. The authors report state-of-the-art performance across several FER benchmarks, including 92.05% accuracy on RAF-DB.

  • Problem

    FER methods have not jointly addressed inter-class similarity, intra-class discrepancy, and scale sensitivity in a unified framework.

    Source: §1 · pp. 1–2

  • Method

    POSTER combines image and landmark streams with transformer-based cross-fusion and a feature pyramid that represents features at multiple scales.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    POSTER outperforms state-of-the-art methods on three commonly used FER datasets; its RAF-DB accuracy is 92.05%.

    Source: §4.3 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The authors present POSTER as a unified framework addressing all three identified FER challenges.

    Source: §1 · p. 2

Abstract

from arXiv · show

Facial expression recognition (FER) is an important task in computer vision, having practical applications in areas such as human-computer interaction, education, healthcare, and online monitoring. In this challenging FER task, there are three key issues especially prevalent: inter-class similarity, intra-class discrepancy, and scale sensitivity. While existing works typically address some of these issues, none have fully addressed all three challenges in a unified framework. In this paper, we propose a two-stream Pyramid crOss-fuSion TransformER network (POSTER), that aims to holistically solve all three issues. Specifically, we design a transformer-based cross-fusion method that enables effective collaboration of facial landmark features and image features to maximize proper attention to salient facial regions. Furthermore, POSTER employs a pyramid structure to promote scale invariance. Extensive experimental results demonstrate that our POSTER achieves new state-of-the-art results on RAF-DB (92.05%), FERPlus (91.62%), as well as AffectNet 7 class (67.31%) and 8 class (63.34%). The code is available at https://github.com/zczcwh/POSTER.

1. Introduction

FER must distinguish subtly different expressions, accommodate varied appearances within an expression, and work across image scales. POSTER combines image and landmark streams through cross-fusion and a feature pyramid to address these challenges together.

1. Introduction

  • FER faces inter-class similarity, intra-class discrepancy, and scale sensitivity, arising respectively from subtle expression differences, varied appearances within a category, and varying image quality or resolution.For example, a small mouth change may distinguish expressions, while images of the same expression can differ in skin tone, age, gender, and background.Source: §1 · p. 1
  • Earlier methods [15] used landmarks but simple concatenation left image-landmark correlations unexplored; recent approaches [30] [42] and addressed only subsets of the challenges.The introduction notes that landmarks had been largely ignored by recent FER approaches.Source: §1 · pp. 1–2
  • POSTER uses image and landmark streams with transformer-based cross-fusion, letting landmarks guide attention to salient regions while image features contribute global information beyond landmarks.The paper motivates landmarks as sparse facial cues less sensitive to appearance variation, while image features cover regions such as cheeks and forehead.Source: §1 · p. 2
  • POSTER integrates a feature pyramid to capture features at multiple resolutions and granularities, addressing scale sensitivity alongside the cross-fusion design.The authors describe the pyramid and cross-fusion transformer as a unified framework for the three FER challenges.Source: §1 · p. 2
  • POSTER outperforms previous state-of-the-art methods on three commonly used datasets, reporting 92.05% on RAF-DB, 67.31% on AffectNet, and 91.62% on FERPlus.Source: §1 · p. 2

2. Related Work

Earlier FER work used landmarks or transformer-based image features, but the cited approaches left landmark–image correlations or scale sensitivity unaddressed. POSTER combines landmark and image streams through cross-fusion and a feature pyramid.

2. Related Work

  • Earlier landmark-based FER combined streams by weighted summation, feature multiplication [15], or landmark-guided cropping [18], while cited SOTA methods [33] [34] [42] omitted landmarks.POSTER is presented as exploring image–landmark correlations while addressing inter-class similarity, intra-class discrepancy, and scale sensitivity together.Source: §2 · pp. 2–3

3.1. Baseline

  • The baseline extracts image and landmark features, concatenates them along the patch dimension, and feeds the fused sequence to vanilla transformer encoders.The image backbone is fine-tuned during training, while the off-the-shelf landmark detector is frozen; the paper says this simple concatenation does not fully exploit feature correlations.Source: §3.1 · p. 3

3.2. POSTER

  • POSTER uses swapped-query cross-fusion so image features gain landmark salient-region guidance while landmark features gain global image context.Separate cross-fusion encoders process large, medium, and small feature levels, whose outputs are aggregated; the design differs from CrossViT-style patch switching.Source: §3.2 · pp. 3, 5

4. Experiments

Experiments evaluate POSTER on RAF-DB, FERPlus, and AffectNet, then examine class-wise performance and architectural choices. The reported comparisons, ablations, and attention visualizations assess its benchmark performance and components.

4.1. Datasets

  • The evaluation covers RAF-DB, FERPlus, and AffectNet, including AffectNet's seven- and eight-category settings.RAF-DB uses seven basic expressions; FERPlus has eight categories, including contempt; AffectNet includes more than one million internet-collected facial images.Source: §4.1 · p. 5

4.2. Implementation Details

  • POSTER uses a finetuned IR50 image backbone and a frozen MobileFaceNet landmark detector, with three pyramid feature levels and eight transformer encoders per level.The large, medium, and small feature embeddings have dimensions 512, 256, and 128, respectively.Source: §4.2 · p. 5

4.3. Comparison with State-of-the-art Results

  • 92.05% RAF-DB accuracy, 91.62% FERPlus accuracy, and AffectNet accuracy of 67.31% (7 classes) and 63.34% (8 classes) lead the reported prior-method comparisons.The reported margins are 1.14% over TransFER on RAF-DB, 0.79% over TransFER on FERPlus, and 1.08% and 2.01% over the second-best methods on AffectNet 7- and 8-class tasks, respectively.Source: Table 1 · p. 6, §4.3 · p. 6

4.4. Results analysis

  • The RAF-DB t-SNE visualization compares baseline and POSTER high-dimensional features; the reported POSTER features show improved class separation and cluster density.Source: Figure 5 · p. 6, §4.4 · p. 7
  • 80% Happy accuracy on AffectNet accompanies a training set where happiness accounts for almost half of images; Fear on RAF-DB and Disgust and Contempt on FERPlus have lower accuracy with fewer samples.AffectNet has 134,415 happiness training images; RAF-DB has 281 Fear images, while FERPlus has 116 Disgust and 120 Contempt images.Source: §4.4 · pp. 6–7
  • POSTER improves class-wise accuracy for most categories across three datasets, and Table 3 reports fewer wrong predictions into certain categories.The authors interpret these patterns as reduced intra-class discrepancy and alleviated inter-class similarity, respectively.Source: Table 2 · p. 7, Table 3 · p. 7, §4.4 · p. 7

4.5. Ablation Study

  • Image-plus-landmark concatenation outperforms either stream alone, cross-fusion further boosts performance, and the pyramid structure improves it; POSTER achieves the best ablation result.The authors attribute pyramid gains to alleviating scale sensitivity across varying input sizes and training-sample quality.Source: Figure 6 · p. 7, Table 4 · p. 7, §4.5 · pp. 7–8
  • On AffectNet images, landmark-stream attention commonly highlights landmark regions but also attends beyond them, while image-stream attention highlights expression-discriminating patterns.Examples include tears for sadness and forehead wrinkles for anger; the authors describe the two streams' attention as different but complementary.Source: Figure 7 · p. 8, §4.5 · p. 8

5. Conclusions

POSTER is a two-stream FER network that uses cross-fusion transformers to exploit image–landmark feature correlations. Experiments on three commonly used datasets show it outperforms state-of-the-art methods.

5. Conclusions

  • POSTER uses cross-fusion transformers to exploit correlations between image and landmark features.Landmarks are detected by an off-the-shelf facial landmark detector, while image features come from a backbone CNN.Source: §5 · p. 8
  • POSTER is a two-stream network with separate landmark and image feature streams.The landmark stream uses an off-the-shelf detector; the image stream uses a backbone CNN.Source: §5 · p. 8
  • Experiments on three commonly used FER datasets show POSTER outperforms state-of-the-art methods.Source: §5 · p. 8

A. Overview

The supplementary material contains additional ablations and analyses, organized into five sections.

A. Overview

  • The supplementary material contains additional ablation studies.Source: A. Overview · p. 9
  • Sections B and C cover pyramid layers and the cross-fusion mechanism.Source: A. Overview · p. 9
  • Sections D, E, and F cover model size and FLOPs, transformer encoder depth, and confusion matrices.Source: A. Overview · p. 9

B. Pyramid Layers:

The pyramid-layer experiments compare three- and four-layer configurations, and POSTER adopts the three-layer configuration considering computational budget.

B. Pyramid Layers

C. Cross-fusion Mechanism

POSTER swaps image and landmark queries for cross-fusion in each transformer attention block; experiments report the best performance when all blocks use this mechanism.

C. Cross-fusion Mechanism

D. Model Size and FLOPs Comparison:

The comparison examines POSTER variants and related methods in terms of model complexity and accuracy. POSTER-T outperforms the image-only case within a similar computational budget, while POSTER with eight cross-fusion transformer blocks achieves the best results.

D. Model Size and FLOPs Comparison:

  • POSTER-T, POSTER-S, and POSTER share the same image backbone and landmark detector, differing only in the number of transformer blocks.Their encoder depths are 4, 6, and 8, respectively.Source: D. Model Size and FLOPs Comparison · p. 9
  • POSTER-T has lower parameter count and similar FLOPs to DMUE while performing better on RAF-DB and AffectNet; it outperforms TransFER on all reported aspects.The comparison includes the image backbone and landmark detector in POSTER's parameter and FLOPs totals.Source: D. Model Size and FLOPs Comparison · p. 9
  • POSTER achieves the best results with 8 cross-fusion transformer blocks, while POSTER-T outperforms image-only features on RAF-DB and AffectNet within a similar computational budget.The image-only transformer baseline reaches 90.51% accuracy on RAF-DB and 65.35% on AffectNet 7cls.Source: D. Model Size and FLOPs Comparison · p. 9

E. Transformer encoders depth:

The depth analysis compares accuracy across transformer encoder counts on RAF-DB and AffectNet. The paper selects depth 8 because it provides the best results.

E. Transformer encoders depth:

  • 8 transformer encoders provide the best results, while performance remains relatively high on RAF-DB and AffectNet when depth exceeds 4.The depth comparison is shown in Fig. 8.Source: E. Transformer encoders depth · p. 9

F. Confusion Matrices:

The confusion matrices compare the baseline and POSTER on RAF-DB and AffectNet 7-class and 8-class settings. POSTER improves class-wise accuracy across all three experiments and reduces wrong-category predictions in most cases.

F. Confusion Matrices:

Loading 2204.04083v2…