AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow

1 Introduction

• The ABAW challenge jointly predicts valence-arousal, eight expressions, and twelve Action Units despite 8× expression and 29× AU imbalance and joint labels on only 37% of training frames.The three targets differ in output type and supervision density.Source: §1 · p. 1
• AffectFlow-DINO adds a conditional rectified-flow head to DINOv3 ViT-S/16, generating plausible joint affect vectors whose mean improves on point estimates and whose spread represents uncertainty.The flow models the full conditional joint distribution p(y | x), motivated by ambiguity from subtle expressions, occlusion, and low-intensity behavior.Source: §1 · pp. 1–2
• The masked rectified-flow MTL objective jointly optimizes deterministic task heads and a generative distribution over the 22-dimensional affect vector, with ablations spanning 26 design choices.The ablations cover flow-loss weight, inference efficiency, imbalance remedies, backbone fine-tuning, and post-hoc calibration.Source: §1 · p. 2
• PMT L = 1.177 on validation, a +0.054 improvement from calibration alone, with Fear F1 rising from 3.8% to 33.1% and Sadness from 17.1% to 28.2%.The calibration applies per-AU and per-class expression thresholds without retraining.Source: §1 · p. 2

ABAW and Aff-Wild lineage

• Aff-Wild/Aff-Wild2 introduced in-the-wild valence-arousal, expression, and Action Unit prediction, while the 7th ABAW established the static-frame three-task format and summed P_MTL metric used by ABAW11 [19–21,38].ABAW8–10 expanded toward multimodal and fine-grained behavior analysis, while ABAW11 retained the VA+EXPR+AU track.Source: §2 · p. 2

Datasets

• s-Aff-Wild2 [11] is the benchmark for this challenge; AffectNet, RAF-DB, DFEW, EmotioNet, BP4D, DISFA, and other related datasets cover partial subsets of the task triad.The related datasets are summarized in Appendix B (Table 7) [1,3, 6,9,25–27,33,34,36,39].Source: §2 · p. 2

MTL for affect

• VA, expressions, and AUs are complementary shared-encoder representations, but incomplete labels, heterogeneous task structures, and negative transfer call for masking, selective fusion, and tuned loss weights [12] [13] [21].Recent ABAW systems combine pretrained MAE or DINOv2 backbones with task-aware fusion, staged training, and temporal modeling [4,5,28,30,35,37].Source: §2 · p. 2

Generative and uncertainty-aware prediction

• Rectified flow transports noise to the joint 22-dimensional affect target in a few steps, addressing multimodal ambiguity that point estimates and scalar-variance heads cannot capture [2].The paper presents this as, to its knowledge, the first application of rectified flow in a heterogeneous multi-task learning setting.Source: §2 · p. 3

SOTA and direct comparisons

• P_MTL = 1.529 is the leading 7th ABAW multi-task leaderboard result, versus the official baseline P_MTL = 0.34; ABAW8/9 use different task splits and are not directly comparable.The leading entry used progressive staged training, task-selective fusion, and temporal context.Source: §2 · p. 3

3.1 Overview

• Since s-Aff-Wild2 provides isolated cropped faces rather than sequences, AffectFlow-DINO is frame-level and jointly predicts VA, expression, and AU targets.The system combines a shared projection, three deterministic task heads, and a conditional rectified-flow head modeling the joint affect distribution.Source: §3.1 · pp. 3–4

3.2 Joint Affect Representation and Masked Supervision

• Each sample’s 22-dimensional joint target comprises VA, an eight-category one-hot expression vector, and 12 binary AUs; per-sample masks exclude unavailable labels from every loss.The flow treats categorical and binary dimensions as real-valued during shared Euler integration, then recovers discrete predictions at inference; intermediate values may be off-manifold. Masked training assumes annotation missingness is independent of true affect, an assumption that may fail for rare or ambiguous labels and bias learned marginals.Source: §3.2 · pp. 4–5, §3.5 · p. 6

3.3 DINOv3 Visual Backbone

• A frozen DINOv3 ViT-S/16 extracts [CLS] features from 224 × 224 face images; its spatially consistent self-supervised tokens suit localized AU deformations [35], unlike MAE.A learned projection maps the backbone features to a 768-dimensional working embedding using LayerNorm and GELU.Source: §3.3 · p. 5

3.4 Deterministic Multi-Task Heads

• Three two-layer MLP heads predict VA, expression, and AU outputs from the shared embedding, using masked cross-entropy for expressions and masked binary cross-entropy for AUs.VA outputs are constrained to the annotation range with tanh; AU probabilities are thresholded at inference.Source: §3.4 · p. 5

3.5 Conditional Rectified Flow

• Rectified-flow training interpolates Gaussian noise and the joint target, then learns their constant transport velocity with a masked squared-error objective conditioned on image features and time.The missing-label mask supervises annotated marginals; under missingness independent of true affect, these updates collectively constrain the full joint distribution, but annotation bias can violate that assumption.Source: §3.5 · p. 6
• AffectFlow-DINO’s final predictions average sampled rectified-flow trajectories, enabling one image to yield an affect distribution and uncertainty estimates alongside official predictions.VA values are clamped, expression labels decoded by argmax, and AU activations by sigmoid thresholding; the AU threshold τ is tunable.Source: §3.5 · p. 7

3.6 Training Objective

• The total objective adds weighted flow loss to deterministic VA, expression, and AU losses; β = 1.0 is the default, and AdamW checkpoints are selected by validation P_MTL.The β ablation found β = 0.5 and β = 1.0 achieve equivalent best validation P_MTL.Source: §3.6 · p. 7

4.1 Experimental Setup

• 36.6% of training frames have joint three-task labels, so per-task masking is essential; all experiments use the official split and a frozen-backbone default configuration.The default uses 20 epochs, 224 × 224 inputs, and, for flow evaluation, N = 16 samples and T = 30 Euler steps; validation selects checkpoints by PMT L.Source: §4.1 · pp. 7–8

4.2 What Does Each Training Objective Contribute?

• PMT L = 0.826 with jointly trained AffectFlow flow decoding exceeds flow-only training at 0.773, while mismatched decode modes collapse to 0.402 and 0.408.Flow decoding improves AffectFlow CCC-V by +0.058 and CCC-A by +0.018 versus its deterministic decode, lifting PVA from 0.200 to 0.238; flow sampling saturates at N = 8 and T = 10.Source: §4.2 · pp. 8–9

4.3 Low-Learning-Rate DINOv3 Fine-Tuning

• PMT L rises from 0.831 to 1.045 with low-learning-rate backbone fine-tuning, where deterministic decoding outperforms flow decoding.Expression loss-weight scaling, focal loss, separate projection heads, and their combination did not improve PEXP R; PSP reached PMT L = 0.859 with a frozen backbone and was not combined with fine-tuning.Source: §4.3 · pp. 9–10

4.4 Per-AU Threshold Calibration

• Per-AU calibration adds approximately +0.06 PAU across checkpoints and gives fine-tuned deterministic decoding PMT L = 1.101 without retraining.Independent validation-set thresholds leave VA and expression predictions unchanged; rare AUs benefit most from lower thresholds, while AU25 and AU7 prefer thresholds near 0.55–0.60.Source: §4.4 · p. 10

4.5 Per-Class Analysis and Class-Weighted Expression Loss

• With inverse-frequency class weighting, after seven training epochs and with a frozen backbone, the best checkpoint reaches PEXP R = 0.239 versus 0.218 for AffectFlow at the same evaluation protocol.Disgust F1 rises from 11.8% to 39.0% and Surprise from 2.8% to 12.8%. A class-balanced WeightedRandomSampler reaches PEXP R = 0.251; AU positive weighting yields PAU = 0.427–0.439 but is incompatible with flow decoding.Source: §4.5 · p. 11

4.6 Flow Retuning after Backbone Fine-Tuning

• PMT L = 1.073 with β=1.0 flow retuning exceeds the β=0.5 result of 1.062, but deterministic decoding still outperforms flow decoding after fine-tuning.Per-AU calibration raises the β=1.0 result to PMT L = 1.123; kitchen-sink fine-tuning reaches PMT L = 1.041 versus 1.045 for plain fine-tuning, and ViT-B and horizontal-flip augmentation do not surpass the best result.Source: §4.6 · pp. 11–12

4.7 Per-Class Expression Threshold Calibration

• Expression calibration raises Fear F1 from 0.038 to 0.331 and Sadness F1 from 0.171 to 0.282 without retraining, while majority-class scores remain essentially unchanged.Validation-tuned per-class logit weights address suppressed minority-class logits; combining this calibration with per-AU calibration yields the best overall result.Source: §4.7 · pp. 12–13

4.8 Summary of Ablation Results

• PMT L = 1.177 is the best overall result, above the official baseline of PMT L = 0.45, after fine-tuning, flow retuning, and combined AU and expression calibration.The score comprises PVA = 0.325, PEXP R = 0.350, and PAU = 0.502; it improves by +0.054 over the per-AU-calibrated flow-retuned baseline.Source: Table 5 · p. 13, §4.7 · p. 13, §4.8 · p. 13

5 Conclusion

• AffectFlow-DINO models in-the-wild facial behavior as a conditional distribution using a rectified-flow head over the joint 22-dimensional affect space.The model was validated through 26 controlled ablations on s-Aff-Wild2.Source: §5 · p. 14
• Flow and deterministic objectives are complementary: flow-only training collapses without a deterministic anchor, while deterministic-only inference loses distributional capability.Flow sampling benefits valence-arousal most and saturates rapidly; backbone adaptation is dominant, but fine-tuning requires joint flow-head retraining.Source: §5 · p. 14
• Post-hoc calibration surfaces rare-class signals, unlocking near-zero F1 for Fear, Sadness, AU15, and AU23 at zero cost.The passage suggests these signals are already encoded in the representation despite severe class imbalance.Source: §5 · p. 14
• Closing the gap to state of the art likely requires temporal aggregation, more expressive fine-tuned decoding, and joint backbone-flow multi-task scaling from the start.Source: §5 · p. 14

• Table 6 gives the unabridged results summary, including intermediate and negative-result configurations omitted from Section 4.8 for space.Source: A Full Results Summary · p. 17

B Related Datasets

• s-Aff-Wild2 is the only benchmark family combining valence-arousal, expression, and AU labels, and it provides them as static image crops.This makes it the natural fit for the challenge.Source: B Related Datasets · pp. 17–18, Table 7 · p. 18
• Table 7 surveys datasets relevant to ABAW-style joint affect modeling, with “All 3” denoting joint availability of valence-arousal, expression, and AU labels.Source: B Related Datasets · p. 17, Table 7 · p. 18

C.1 Flow Loss Weight

• Performance is stable for β ∈ [0.5, 1.0], but P_MTL drops at β = 2.0; the adopted default is β = 1.0.The sweep holds other hyperparameters fixed and uses flow decoding with N = 16 and T = 30.Source: Table 8 · p. 18, §C.1 · p. 18
• The flow loss-weight sweep tests β ∈ {0.25, 0.5, 1.0, 2.0} with other hyperparameters fixed and flow decoding at N = 16, T = 30.Source: §C.1 · p. 18

C.2 Inference Efficiency

• N = 1 already outperforms the official baseline (PMT L = 0.45), consistent with rectified flow's straight-line trajectory property; the default is N = 16, T = 30.Source: §C.2 · p. 18
• Sampling saturates at N = 8, and all tested integration step counts are within 0.007 PMT L of each other.Source: §C.2 · p. 18, Table 9 · p. 19

C.3 PSP: Patch Soft-Pool Feature Aggregation

• PSP Det reaches PMT L = 0.859, the best frozen-backbone result without calibration, +0.033 over AffectFlow flow, with gains in all three tasks.The comparison uses a frozen DINOv3 ViT-S/16 backbone trained for 20 epochs.Source: Table 10 · p. 19, §C.3 · p. 19
• PSP learns per-channel attention over 196 spatial patch tokens and concatenates the pooled patch features with the DINOv3 CLS token before projection.Source: §C.3 · p. 19
• PSP attention weights use a softmax over learned weights, start uniformly, and expand projection input to 2d = 768; only pooling weights are trained with the backbone frozen.Source: §C.3 · p. 19
• PSP Flow underperforms AffectFlow Flow (0.801 vs. 0.826), likely because its doubled input dimension requires longer flow-head retuning; PSP with backbone fine-tuning remains future work.Source: §C.3 · p. 19

D ViT-B Backbone and Test-Time Augmentation

• ViT-B reaches CCC-V 0.416 vs. 0.387 and per-AU calibrated score 0.507 vs. 0.502, but its PMT L = 1.116 is below the ViT-S best (1.123).Its PEXP R decreases by 0.041; horizontal-flip TTA hurts all three sub-scores, with altered asymmetric AU patterns offered as an explanation.Source: D ViT-B Backbone and Test-Time… · p. 19, Table 11 · p. 20

E Additional Ablation Studies

• The additional ablations cover VA loss weight, global AU threshold, expression enhancement, label smoothing, and global AU loss weight, but none improves over the AffectFlow baseline.Source: E Additional Ablation Studies · p. 20

E.1 VA Loss Weighting

• Arousal estimation is the weakest component across runs, motivating tests of VA loss weights λV A ∈{1, 2, 4} for improving arousal CCC without harming other tasks.Source: §E.1 · p. 20
• λV A = 2 yields the best PMT L with a marginal +0.004 gain, while λV A = 4 improves arousal but reduces expression performance and the overall score; the default λV A = 1 is retained.The λV A = 2 result improves PEXP R without harming PV A, but the gain is not consistent across ablations; the stronger weighting is linked to negative transfer at extreme task-weight imbalance.Source: §E.1 · p. 20

E.2 Global AU Threshold Sweep

• A global AU threshold sweep on the AffectFlow checkpoint does not improve on the default 0.5 threshold; lower thresholds increase false positives, and 0.7 sharply reduces AU F1.The sweep uses flow decoding with N = 16 samples and T = 30 steps.Source: §E.2 · p. 20, Table 13 · p. 21
• A single global threshold is too coarse for heterogeneous AU imbalance, so per-AU calibration is necessary.Source: §E.2 · p. 21

E.3 Expression Enhancement Failures

• Expression loss-weight scaling and focal loss fail to improve PEXP R; focal loss also reduces PV A by approximately 0.016.The passages attribute the failures to majority-class-dominated cross-entropy and too few valid expression labels (90K training frames) under heavy class skew.Source: §E.3 · pp. 21–22, Table 14 · p. 22, Table 15 · p. 22
• Separate task projections slightly reduce all three sub-scores, while combining them with focal loss improves PEXP R by only +0.003 over separate projections and remains below the AffectFlow baseline.These results suggest that, with a frozen backbone, shared projections regularize rather than bottleneck the shared representation.Source: §E.3 · pp. 22–23, Table 16 · p. 23, Table 17 · p. 23

E.4 Label Smoothing

• Mild label smoothing (ϵ = 0.05) raises PEXP R to 0.230 and CCC-V to 0.276, but at ϵ = 0.20, PEXP R falls below the unsmoothed baseline.Label smoothing alone is insufficient to close the expression imbalance gap.Source: §E.4 · p. 23, Table 18 · p. 24

E.5 Global AU Loss Weight

• Neither reducing nor increasing λAU changes PMT L appreciably; both variants plateau at 0.819.Per-AU positive weighting achieves a larger PAU gain (+0.043) than any global weight setting.Source: §E.5 · p. 23, Table 19 · p. 24

F Class Distribution Statistics

• Table 20 provides the full per-class expression counts and AU positive rates referenced throughout the paper.Source: F Class Distribution Statistics · p. 23

G.1 Balanced Expression Sampling

• Balanced sampling raises PEXP R to 0.251 (+0.039 over AffectFlow flow) and lifts PMT L to 0.864, the best frozen-backbone result for expression.Adding class-weighted cross-entropy gives PEXP R = 0.246 under deterministic decode but drops PV A to 0.193 vs. 0.238.Source: §G.1 · p. 24

G.2 BCE Positive Weighting for AUs

• Flow decoding degraded both weighted variants, yielding PMT L = 0.777 for AU weighting alone and 0.745 for combined weighting.The reweighted loss shifted predicted probability distributions incompatibly with the rectified-flow sampler.Source: §G.2 · p. 24
• PAU rose from 0.384 to 0.427 (+0.043) with AU positive weighting, then to 0.439 with added expression class weighting.AU BCE positive weights were set to nneg/npos from the training set.Source: §G.2 · p. 24, Table 20 · p. 25
• PMT L = 0.890 was the best frozen-backbone result without calibration, marginally surpassing AffectFlow after per-AU calibration (0.888).The result followed adding expression class weighting to AU positive weighting.Source: §G.2 · pp. 24–25

H Kitchen-Sink Fine-Tuning

• Expression macro-F1 is 0.288 with kitchen-sink fine-tuning versus 0.285 with plain fine-tuning, showing no meaningful expression benefit from AU positive weighting.The prose attributes the AU gain and offsetting VA drop to gradient competition when the backbone is jointly optimized.Source: H Kitchen-Sink Fine-Tuning · pp. 25–26
• PMT L is 1.041 with kitchen-sink fine-tuning versus 1.045 with plain fine-tuning, as PAU rises from 0.441 to 0.483 while PV A falls from 0.318 to 0.271.The combination applies class-weighted cross-entropy and per-AU BCE positive weighting during the same low-LR fine-tuning run.Source: H Kitchen-Sink Fine-Tuning · p. 25, Table 23 · p. 26
• After per-AU calibration, kitchen-sink fine-tuning reaches PMT L = 1.061, below the best overall flow-retuned (β=1.0) calibrated result of 1.123.The results indicate that frozen-backbone reweighting strategies add little value after fine-tuning and can harm PV A; flow retuning is more effective.Source: Table 23 · p. 26, H Kitchen-Sink Fine-Tuning · p. 26

I Per-Class and Per-AU F1 Breakdowns

• Table 24 reports per-class expression F1 for AffectFlow (flow, N=16) and the fine-tuned backbone (deterministic).Table 25 reports fine-tuned-model per-AU F1 at threshold 0.5 and after per-AU calibration.Source: I Per-Class and Per-AU F1 Breakdowns · p. 26, Table 24 · p. 27, Table 25 · p. 28
• Fear F1 rises from 0.038 to 0.331 (+0.293) and Sadness from 0.171 to 0.282 (+0.111) after expression calibration, while macro F1 rises from 0.296 to 0.350.These results are for the fine-tuned+flow retune (β=1.0) checkpoint.Source: Figure 3 · p. 27, Table 26 · p. 28
• AU15 and AU23, with positive rates of 2.4% and 2.9%, recover from near-zero default-threshold F1 to 10.8% and 18.9% after per-AU calibration.The passage interprets this recovery as evidence that the model learned discriminative signal suppressed by the imbalanced training objective.Source: I Per-Class and Per-AU F1 Breakdowns · p. 26, Table 25 · p. 28