PaperlayerView exact revision on arXiv

Source-linked AI summary

AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow

Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika, Abdenour Hadid

arXiv:2607.13250v1cs.CV

TL;DR

In-the-wild facial behavior is ambiguous, while multi-task affect benchmarks combine heterogeneous targets with severe imbalance and incomplete joint annotation. AffectFlow-DINO models a conditional affect distribution with rectified flow, achieving P_MTL=1.177 versus the official baseline’s 0.45 on its validation split.

  • Problem

    In-the-wild affect estimation must handle perceptual ambiguity alongside heterogeneous targets, severe imbalance, and incomplete joint annotation.

  • Method

    AffectFlow-DINO jointly predicts valence-arousal, expressions, and Action Units while learning their conditional joint distribution with a rectified-flow head.

  • Results

    P_MTL=1.177 substantially improves over the official baseline P_MTL=0.45 on the local s-Aff-Wild2 validation split.

  • Takeaways & Limitations

    Backbone adaptation is the dominant performance lever, but the flow head must be retrained jointly during fine-tuning to preserve generative capability.

  • Takeaways & Limitations

    Evaluation uses only the local s-Aff-Wild2 validation split, whose results are not directly comparable with cited hidden-test SOTA scores.

Abstract

from arXiv · show

We present \textbf{AffectFlow-DINO}, a multi-task learning system for the 11th ABAW challenge that extends a standard deterministic architecture with a conditional rectified-flow head to model the inherent ambiguity of in-the-wild facial behavior. Instead of predicting a single affect estimate, the model learns a conditional generative distribution, enabling uncertainty-aware one-to-many predictions through Monte Carlo sampling. The system jointly estimates continuous valence-arousal, classifies eight facial expressions, and detects twelve Action Units from static face images. Built on a frozen DINOv3 ViT-S/16 backbone, extensive ablation studies show that rectified-flow decoding consistently improves deterministic prediction, particularly for valence-arousal estimation (CCC-V $+0.058$). We further show that post-hoc threshold calibration effectively recovers performance on severely imbalanced rare classes (e.g., Fear: $3.8\% \rightarrow 33.1\%$) without retraining. Combined with backbone fine-tuning and flow retuning, the final model achieves $\mathbf{P_{MTL}=1.177}$, substantially outperforming the official challenge baseline of $P_{MTL}=0.45$.

1 University of the Basque Country (UPV/EHU), Spain

The section identifies the paper’s international institutional affiliations and its focus on affective behavior analysis using multi-task learning and rectified flow.

  • Affiliations: The listed affiliations include the University of Salento, CNR’s Institute of Applied Sciences and Intelligent Systems, and Universiti Malaysia Kelantan.These institutions are located in Italy and Malaysia.
  • Keywords: The stated keywords center on affective behavior analysis, ABAW, multi-task learning, and rectified flow.These terms define the paper’s methodological and application context.
  • Keywords: The task-specific keywords are valence-arousal estimation, facial expression recognition, and action unit detection.Together, they specify the affect-analysis outputs addressed by the work.

1 Introduction

AffectFlow-DINO addresses the ABAW multi-task setting by combining deterministic task heads with a conditional rectified-flow model for ambiguous joint affect predictions. Its ablations and post-hoc calibration target design efficiency and severe imbalance, achieving PMT L = 1.177 on validation.

  • Challenge setting: The ABAW MTL challenge jointly predicts continuous valence-arousal, eight-way expressions, and twelve binary Action Units from each facial frame.The targets differ in output type and supervision density, while s-Aff-Wild2 has severe expression-class imbalance.
  • AffectFlow-DINO: AffectFlow-DINO augments a DINOv3 ViT-S/16 backbone with a conditional rectified-flow head for frame-level multi-task learning.The method uses deterministic task heads alongside a generative conditional distribution over the 22-dimensional affect vector.
  • AffectFlow-DINO: Rectified flow models p(y | x) as a generative transport from noise, preserving uncertainty that deterministic point estimates discard under ambiguous facial behavior.The motivation includes subtle smiles, partial occlusion, and low-intensity expressions with multiple plausible joint VA, expression, and AU configurations.
  • Experimental scope: The study evaluates 26 design choices spanning flow weighting, inference efficiency, imbalance remedies, backbone fine-tuning, and post-hoc calibration.The ablation study is conducted on s-Aff-Wild2.
  • Results: PMT L = 1.177 on the validation split, with calibration alone improving the score by +0.054.Calibration recovered Fear F1 from 3.8% to 33.1% and Sadness from 17.1% to 28.2%.

2 Related Work

Prior work established ABAW as a static-frame, three-task affect benchmark spanning valence-arousal, expressions, and Action Units, while motivating task-aware multi-task learning and generative uncertainty modeling. The 7th ABAW leaderboard provides the closest comparison, with a leading score of 1.529 versus the official baseline of 0.34.

  • ABAW and Aff-Wild lineage: ABAW evolved from Aff-Wild/Aff-Wild2 into a joint static-frame benchmark for continuous valence-arousal, categorical expressions, and binary Action Units.The ABAW1–6 competitions expanded dataset scale, task coverage, and evaluation protocols.
  • Datasets: s-Aff-Wild2 is the challenge benchmark, while AffectNet, RAF-DB, DFEW, EmotioNet, BP4D, and DISFA cover partial subsets of the task triad.These related datasets are summarized in Appendix B, Table 7.
  • MTL for affect: Shared encoders can jointly learn VA, expressions, and AUs, but label incompleteness, heterogeneous task structures, and negative transfer require masking, selective fusion, and tuned loss weights.Recent ABAW solutions also combine pretrained MAE or DINOv2 backbones with task-aware fusion and staged training.
  • Generative and uncertainty-aware prediction: Generative transports over label space can outperform deterministic regressors for inherently multimodal targets, whereas Gaussian and evidential VA heads cannot represent joint multimodal affect ambiguity.This motivates generative and uncertainty-aware prediction beyond scalar-variance extensions.
  • SOTA and direct comparisons: 1.529 is the leading 7th ABAW MTL leaderboard score, compared with the official challenge baseline of 0.34.The leading entry uses progressive staged training, task-selective fusion, and temporal context; ABAW8/9 are not directly comparable because they use different task splits.

3 Method

AffectFlow-DINO is a frame-level multi-task model that combines deterministic affect heads with a conditional rectified-flow head for uncertainty-aware predictions. It unifies valence-arousal, expressions, and Action Units in a masked 22-dimensional target space and decodes predictions through sampling and task-specific transformations.

  • Architecture: AffectFlow-DINO uses a frozen DINOv3 ViT-S/16 backbone, shared affect projection, three deterministic task heads, and a conditional rectified-flow head.The heads address valence-arousal regression, expression classification, and Action Unit detection, while the flow head models the joint affect distribution conditioned on image features.
  • Conditional rectified flow: Rectified flow learns a conditional generative distribution from Gaussian noise, enabling multiple plausible affect predictions for each ambiguous face image.At inference, independent noise samples are propagated with Euler integration, averaged, and converted to task outputs using clamping, argmax, and sigmoid thresholding.
  • Target representation: The model represents valence-arousal, eight expressions, and twelve Action Units as a unified 22-dimensional affect target.Valence and arousal are continuous, expressions are one-hot categorical targets, and Action Units are binary targets.
  • Masked learning: Masked supervision ensures missing annotations contribute neither gradients nor distortion to deterministic or flow training signals.The validity mask is applied to every loss term, and the flow formulation supervises only annotated marginals under an explicit missingness assumption.
  • Optimization: The total objective combines weighted deterministic multi-task loss with flow loss, with β = 1.0 adopted after validation ablation.The ablation tested β ∈{0.25, 0.5, 1.0, 2.0}; β ∈{0.5, 1.0} achieved equivalent best validation PMT L.

4 Experiments

Experiments show that jointly trained deterministic and rectified-flow objectives improve affect estimation, with flow sampling especially benefiting continuous valence-arousal. Backbone fine-tuning, flow retuning, and post-hoc AU and expression calibration culminate in PMT L = 1.177.

  • Training-objective ablation: Joint training is necessary: flow-only training with flow decoding reaches PMT L = 0.773, while jointly trained AffectFlow reaches PMT L = 0.826.Mismatched training and decoding collapse to PMT L = 0.402 and PMT L = 0.408.
  • Training-objective ablation: +0.058 CCC-V and +0.018 CCC-A result from flow decoding trained AffectFlow, lifting PV A from 0.200 to 0.238.Continuous targets benefit most from distributional averaging, while categorical and binary tasks benefit less.
  • Backbone adaptation: 1.045 PMT L is achieved through backbone fine-tuning, rising from 0.831, although deterministic decoding then outperforms flow decoding.The results suggest the adapted backbone strengthens deterministic heads while the flow head may require retuning.
  • Flow retuning and calibration: 1.073 PMT L follows flow retuning at β=1.0 with deterministic decoding; per-AU calibration raises this to 1.123, and expression calibration further reaches 1.177.At β=1.0, CCC-V is 0.387 versus 0.308 for β=0.5.
  • Flow retuning and calibration: 0.325 PV A + 0.350 PEXP R + 0.502 PAU = 1.177 PMT L is the best overall calibrated result.This improves by +0.054 over the flow-retuned, per-AU-calibrated baseline of 1.123.

5 Conclusion

AffectFlow-DINO models in-the-wild facial behavior as a conditional distribution over a joint 22-dimensional affect space, combining rectified-flow and deterministic objectives. The conclusion highlights complementary training objectives, the importance of joint backbone-flow adaptation, post-hoc calibration for rare classes, and directions for closing the SOTA gap.

  • Conclusion: AffectFlow-DINO learns p(y | x) with a rectified-flow head over the joint 22-dimensional affect space, validated through 26 controlled ablations.The model treats facial behavior as a conditional distribution rather than a deterministic mapping.
  • Conclusion: Flow-only training collapses without a deterministic anchor, while deterministic-only inference discards distributional capability.The two objectives are complementary rather than interchangeable.
  • Conclusion: Backbone adaptation is the dominant lever, but the flow head must be retrained jointly during fine-tuning.Flow sampling benefits valence-arousal estimation most and saturates rapidly, consistent with rectified flow’s straight-line transport.
  • Conclusion: Post-hoc calibration unlocks near-zero F1 for rare categories including Fear, Sadness, AU15, and AU23 without retraining cost.This suggests that calibration can surface rare-class signal already encoded in the representation despite severe imbalance.
  • Conclusion: Closing the gap to SOTA likely requires temporal aggregation, more expressive fine-tuned decoding, and joint backbone-flow multi-task scaling from the start.These are identified as the primary directions for future improvement.

A Full Results Summary

Table 6 provides the complete results summary for Section 4.8, including intermediate and negative-result configurations omitted from the main paper for space.

  • A Full Results Summary: Table 6 gives the unabridged Section 4.8 results summary, restoring intermediate and negative-result configurations excluded from the main paper.These configurations were omitted from the main paper because of space constraints.

B Related Datasets · C Flow Weight, Inference Efficiency, and Patch Soft-Pool · C.1 Flow Loss Weight

The related-dataset review identifies s-Aff-Wild2 as the only benchmark combining all three affect tasks in static image crops. A flow-loss sweep finds stable performance for β ∈[0.5, 1.0], with β = 1.0 adopted as default because β = 2.0 reduces PMT L.

  • B Related Datasets: Table 7 surveys datasets relevant to ABAW-style joint affect modeling.The table concerns joint valence-arousal, expression, and Action Unit annotation availability.
  • B Related Datasets: s-Aff-Wild2 is the only benchmark combining all three task targets.
  • B Related Datasets: s-Aff-Wild2 provides valence-arousal, expression, and AU labels as static image crops, making it a natural fit for the challenge.
  • B Related Datasets: “All 3” denotes joint availability of valence-arousal, expression, and AU labels within the same benchmark family.
  • C.1 Flow Loss Weight: The flow-loss sweep evaluates β ∈{0.25, 0.5, 1.0, 2.0} with fixed hyperparameters and inference decoding at N = 16, T = 30.
  • C.1 Flow Loss Weight: Performance remains stable for β ∈[0.5, 1.0], whereas β = 2.0 causes the flow objective to dominate and PMT L to drop.The default setting is β = 1.0.

C.2 Inference Efficiency … E.2 Global AU Threshold Sweep

The ablations identify efficient rectified-flow inference, patch soft-pooling, backbone choice, and calibration limits across affect tasks. PSP improves frozen-backbone deterministic performance, while ViT-B, extreme VA weighting, and global AU thresholds introduce task-specific trade-offs or fail to improve the baseline.

  • C.2 Inference Efficiency: N = 8 saturates sampling, all integration-step settings remain within 0.007 PMT L, and N = 1 already exceeds the official baseline; N = 16, T = 30 is used by default.The official baseline is PMT L = 0.45.
  • C.3 PSP: Patch Soft-Pool Feature Aggregation: PSP learns per-channel softmax attention over 196 patch tokens, concatenates pooled patches with the CLS token, and doubles the projection input to 2d = 768 while freezing the backbone.Only the pooling weights are trained, with weights initialized uniformly.
  • C.3 PSP: Patch Soft-Pool Feature Aggregation: 0.859 PMT L makes PSP Det the best uncalibrated frozen-backbone result, while PSP Flow reaches 0.801 versus AffectFlow Flow at 0.826.PSP Det improves by +0.033 over the AffectFlow flow result and gains consistently across all three tasks.
  • D ViT-B Backbone and Test-Time Augmentation: 0.416 CCC-V and 0.507 per-AU calibrated score improve with ViT-B versus 0.387 and 0.502, but PEXP R falls by 0.041, producing 1.116 calibrated PMT L below ViT-S’s 1.123.Horizontal-flip TTA hurts all three sub-scores, potentially because flips alter asymmetric AU patterns.
  • E Additional Ablation Studies: Additional ablations covering VA weighting, global AU thresholds, expression enhancement, label smoothing, and global AU loss weighting did not improve the AffectFlow baseline.These experiments are collected as unsuccessful alternatives in the appendix.
  • E.1 VA Loss Weighting: +0.004 PMT L is the marginal gain from λV A = 2, while λV A = 4 improves arousal but reduces expression performance and overall score; λV A = 1 remains the default.The λV A = 2 gain is not consistent across ablations.
  • E.2 Global AU Threshold Sweep: 0.7 sharply reduces AU F1, and no single global threshold improves over the default 0.5 because lower thresholds increase false positives.The result indicates that heterogeneous AU imbalance requires per-AU rather than global calibration.

E.3 Expression Enhancement Failures … G.1 Balanced Expression Sampling

Expression-focused loss and architectural modifications consistently failed to improve performance, while label smoothing offered only a small, unstable benefit. Balanced expression sampling provided the strongest frozen-backbone remedy, whereas global AU loss scaling had little effect.

  • E.3 Expression Enhancement Failures: Expression-specific modifications consistently degraded PV A and failed to improve PEXP R over the AffectFlow baseline.The tested modifications included expression-loss weighting, focal loss, separate projection heads, and their combination.
  • E.3 Expression Enhancement Failures: Focal loss reduced PV A by approximately 0.016 without improving PEXP R under heavy class skew and limited valid expression labels.The focal modulator suppressed majority-class gradients without supplying enough rare-class signal to offset lost VA supervision.
  • E.4 Label Smoothing: Label smoothing reached PEXP R = 0.230 and CCC-V = 0.276 at ϵ = 0.05, but stronger smoothing erased the gain.At ϵ = 0.20, PEXP R fell below the unsmoothed baseline, leaving expression imbalance unresolved.
  • E.5 Global AU Loss Weight: Global AU loss-weight variants both plateaued at PMT L = 0.819, while per-AU positive weighting achieved a larger PAU gain of +0.043.This comparison supports per-class reweighting over global loss scaling for AU detection.
  • F Class Distribution Statistics: Table 20 reports the full per-class expression counts and AU positive rates used throughout the paper.These statistics provide the class-distribution reference for the imbalance experiments.
  • G.1 Balanced Expression Sampling: WeightedRandomSampler rebalanced expression classes by assigning each frame a probability inversely proportional to its expression-class count.The method changed the effective sampling distribution without modifying the loss function.
  • G.1 Balanced Expression Sampling: PEXP R reached 0.251 (+0.039 over AffectFlow flow) and PMT L reached 0.864 with balanced sampling, the best frozen-backbone expression result.Adding class-weighted cross-entropy instead produced PEXP R = 0.246 under deterministic decoding, while PV A fell to 0.193 versus 0.238.

G.2 BCE Positive Weighting for AUs

AU positive weighting improves AU performance, and adding expression class weighting raises it further. However, flow decoding degrades reweighted runs, while the combined strategy achieves the best frozen-backbone result without calibration.

  • AU positive weighting: P_MTL = 0.777 for AU positive weighting alone and 0.745 for the combined variant under flow decoding, which degrades both runs.The reweighted loss shifts predicted probability distributions incompatibly with rectified-flow sampling.
  • AU positive weighting: AU positive weighting raises P_AU from 0.384 to 0.427 (+0.043) at training time.Weights are set per AU as nneg/npos from the training set.
  • AU positive weighting: Adding expression class weighting further lifts P_AU to 0.439.This combines AU-head positive weighting with class-weighted expression cross-entropy.
  • AU positive weighting: P_MTL = 0.890 is the best frozen-backbone result without calibration, marginally surpassing AffectFlow after per-AU calibration (0.888).All variants use a frozen backbone for 20 epochs, with decode mode matched to best-checkpoint evaluation.

H Kitchen-Sink Fine-Tuning · I Per-Class and Per-AU F1 Breakdowns

Kitchen-sink fine-tuning improves AU performance but lowers valence-arousal and does not surpass plain fine-tuning or flow retuning. Per-AU and per-class threshold calibration substantially recovers rare-label F1, especially for AU15, AU23, and Fear.

  • H Kitchen-Sink Fine-Tuning: Class-weighted cross-entropy and per-AU positive weighting are applied jointly during the same low-LR fine-tuning run.The evaluation reports deterministic predictions with and without per-AU calibration.
  • H Kitchen-Sink Fine-Tuning: 1.041 vs. 1.045: Kitchen-sink fine-tuning does not improve PMT L over plain fine-tuning without calibration.PAU rises from 0.441 to 0.483, but PV A falls from 0.318 to 0.271.
  • H Kitchen-Sink Fine-Tuning: 1.061: Per-AU calibration raises the kitchen-sink variant’s PMT L, but it remains below the calibrated flow-retuned result of 1.123.The results indicate that frozen-backbone class-reweighting strategies add little value after fine-tuning and can harm PV A.
  • I Per-Class and Per-AU F1 Breakdowns: 10.8% and 18.9%: Per-AU calibration recovers AU15 and AU23 F1 from near-zero default-threshold values.These are the two rarest AUs, with positive rates of 2.4% and 2.9%.
  • I Per-Class and Per-AU F1 Breakdowns: 0.038 → 0.331: Fear gains the most from per-class threshold calibration, followed by Sadness at 0.171 → 0.282.The results are reported for the fine-tuned+flow retune (β=1.0) checkpoint.
  • I Per-Class and Per-AU F1 Breakdowns: 0.296 to 0.350: Per-class expression calibration improves macro F1 on fine-tuned+flow retune deterministic predictions.Table 26 compares default arg max predictions with calibrated predictions.
Loading 2607.13250v1…