Source-linked AI summary
Self-supervised learning of a facial attribute embedding from video
Olivia Wiles, A. Sophia Koepke, Andrew Zisserman
TL;DR
Facial-attribute prediction is difficult to supervise because labels can be costly or ambiguous, and separate tasks fail to exploit related facial cues. FAb-Net learns an embedding by transforming frames from the same video face-track, then performs comparably or better than self-supervised methods across facial-attribute tasks while approaching supervised performance.
Problem
Facial-attribute labels can be expensive or difficult to obtain in unconstrained settings, and separately trained tasks do not leverage the relationships among landmarks, pose, and expression.
Source: §1 · p. 1Method
FAb-Net learns a low-dimensional embedding without hand labels by predicting a target frame from a source frame in the same face-track.
Source: §1 · p. 2Results
FAb-Net is comparable or superior to other self-supervised methods across landmark, pose, and expression tasks, approaching supervised-method performance.
Source: §1 · p. 3Takeaways & Limitations
The learned embedding supports prediction of pose, landmarks, and expression on other datasets despite domain shift, using only a trained linear layer.
Source: §6 · p. 10Using additional source frames requires additional encoder forward passes, increasing memory requirements unless enough GPU memory allows parallel processing.
Source: §5.2 · p. 9
Abstract
from arXiv · showhide
We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time. To perform this task, we introduce a network, Facial Attributes-Net (FAb-Net), that is trained to embed multiple frames from the same video face-track into a common low-dimensional space. With this approach, we make three contributions: first, we show that the network can leverage information from multiple source frames by predicting confidence/attention masks for each frame; second, we demonstrate that using a curriculum learning regime improves the learned embedding; finally, we demonstrate that the network learns a meaningful face embedding that encodes information about head pose, facial landmarks and facial expression, i.e. facial attributes, without having been supervised with any labelled data. We are comparable or superior to state-of-the-art self-supervised methods on these tasks and approach the performance of supervised methods.
1 Introduction
FAb-Net asks whether video can teach a facial-attribute embedding without hand labels, using frame transformation as a proxy task. The resulting embedding supports several facial-attribute tasks and performs comparably or better than other self-supervised methods, approaching supervised methods.
1 Introduction
- Predicting head pose and expression matters for systems that interact with people and respond to changes in emotional state.The introduction also connects facial-expression understanding to how babies and children perceive people they interact with.Source: §1 · p. 1
- Supervised attribute methods need substantial labeled data, while separate tasks miss the links among landmarks, pose, and expression.The passage notes that three annotators agreed on emotion categories only 46% of the time.Source: §1 · p. 1
- The multi-source variant predicts confidence maps to weight source-frame contributions, and the paper also introduces curriculum training to improve the embedding.The overview describes confidence maps for weighting source frames; the introduction identifies both multi-frame use and curriculum learning as contributions.Source: Figure 1 · p. 2, §1 · pp. 2–3
- FAb-Net learns a low-dimensional facial-attribute embedding by transforming a source frame to a target frame from the same video face-track, without hand labels.The proxy task requires the embeddings to retain information needed for the transformation, including head pose and expression.Source: §1 · p. 2
- Across landmark, pose, and expression tasks, FAb-Net is comparable or superior to self-supervised methods and approaches supervised-method performance.The embedding is used by training a linear layer; qualitative retrieval also groups images with similar facial attributes across identities.Source: §1 · p. 3
2 Related Work
Related work uses image transformations, video, and labeled or weakly labeled face data to learn representations. FAb-Net instead treats frame synthesis as a proxy task and learns an unconstrained embedding from unordered video frames.
2 Related Work
- Self-supervised methods use image data as proxy supervision, with features learned from abundant data transferable to tasks or domains with fewer annotations.Examples transform images to recognize rotations or learn equivariant pixel embeddings.Source: §2 · p. 3
- Video methods include frame-synthesis approaches, but FAb-Net treats synthesis only as a proxy and uses unordered frames rather than temporal order or future-frame prediction.Other cited approaches predict motion explicitly or condition generation on another modality such as voice.Source: §2 · p. 3
- Unlike landmark-reconstruction methods such as Zhang et al. and Jakab et al. [21], FAb-Net imposes no explicit landmark-heatmap representation.Other face embeddings use labeled attributes, 3D morphable models, or metadata such as object labels and weather or geolocation.Source: §2 · pp. 3–4
3 Method
FAb-Net learns facial attributes by generating one frame from another within the same face-track. It embeds each frame, predicts source-to-target sampling, and extends the setup to multiple sources with confidence maps.
3 Method
- Frames from the same face-track are used as source and target so the network learns an embedding that encodes facial attributes without labels.The broader method trains the network to transform one or more source frames into the target frame.Source: §3 · p. 4
3.1 Single-source frame architecture
- Shared-weight encoders map each frame to a 256-dimensional embedding; a decoder uses the concatenated embeddings to predict sampling offsets and generate the target from source pixels.Training minimizes the L1 difference between the generated and target frames; the decoder's shared information path makes the target embedding carry pose and expression cues needed for sampling.Source: §3.1 · p. 4
3.2 Multi-source frames architecture
- For multiple sources, per-source confidence heatmaps estimate flow certainty at each pixel and are combined pixel-wise with a softmax.The maps can express uncertainty for image regions affected by pose or expression differences between source and target frames.Source: §3.2 · pp. 4–5
4 Curriculum Strategy
FAb-Net is trained through progressively harder examples, using per-pair loss to rank samples and validation plateaus to advance the training subset.
4 Curriculum Strategy
- Progressively harder samples proved crucial, while training only on the hardest samples stopped the network from learning.Sample difficulty is ranked by the loss from a forward pass.Source: §4 · p. 5
- Training begins with the lowest-loss 50% of source-target pairs, then shifts the selected percentile window upward by 10 when validation loss plateaus.After four shifts, training uses the 40th–90th percentile range.Source: §4 · p. 5
- The curriculum ends at the 40th–90th percentile because the hardest 10% may be too challenging or contain problematic background shifts.Source: §4 · p. 5
5 Experiments
The experiments test FAb-Net’s representation on landmark, pose, and expression prediction, and examine multi-source behavior and cross-identity image retrieval. Results are competitive with or better than self-supervised methods and approach supervised performance.
5 Experiments
- FAb-Net is evaluated on VoxCeleb+ video data, with identities split into 75/15/10 train, validation, and test sets and frames sampled at 1 fps.Frames are cropped and resized to 256 × 256; training uses SGD, with batch size 32 under curriculum and 8 otherwise.Source: §5 · p. 5
5.1 Using the embedding for regression and classification
- For landmark, pose, and expression tasks, a batch-normalization layer and linear layer are trained on target-dataset labels while the FAb-Net encoder stays fixed.Regression uses MSE and classification uses cross-entropy.Source: §5 · p. 5, §5.1 · p. 6
5.1.1 Baselines
- Baselines include self-supervised models trained on VoxCeleb+ and a VGG-Face descriptor whose training requires labeled face images.The VGG-Face descriptor provides a strong baseline for expression and, to some extent, pose prediction.Source: §5.1.1 · p. 6
5.1.2 Results
- FAb-Net is competitive with or better than self-supervised landmark methods and outperforms self-supervised expression methods on both datasets, approaching supervised performance.Landmark prediction is compared on MAFL and 300-W; expression classification is evaluated on EmotioNet and AffectNet.Source: §5.1.2 · pp. 6, 8, 9, Table 1 · p. 7, Table 3 · p. 8
- On AFLW, FAb-Net predicts roll better than the supervised baseline and achieves comparable MAE, while outperforming the identity-trained VGG-Face descriptor.The embedding is used for linear pose regression; the test set excludes the 1,000 images used in.Source: §5.1.2 · p. 7, Table 2 · p. 7
- Expression evaluation uses in-the-wild AffectNet and EmotioNet data; AUC is computed per class and averaged, with underrepresented training classes reweighted.The datasets use held-out validation subsets because their challenge test sets are unreleased.Source: §5.1.2 · p. 8
5.2 What is the benefit of additional source frames?
- Additional source frames improve performance but require two extra encoder passes; confidence heatmaps select pose-similar sources and can use different frames for different face parts.With sufficient GPU memory, the passes can run in parallel, increasing memory requirements without affecting computational speed.Source: §5.2 · p. 9, Figure 4 · p. 9
5.3 Image retrieval
- Across identities, cosine-similarity retrieval returns images with poses and expressions similar to the query, while remaining largely unaffected by identity and facial decorations.The retrieval test ranks 10,000 randomly sampled VoxCeleb+ test images.Source: §5.3 · pp. 9–10, Figure 5 · p. 10
6 Conclusion
FAb-Net learns facial attributes from videos without hand labels, and its embeddings support pose, landmark, and expression prediction on other datasets. Its performance is competitive with or superior to supervised and self-supervised methods across several tasks.
6 Conclusion
- FAb-Net is competitive with or superior to supervised and self-supervised methods on pose regression and facial landmark detection, and approaches supervised performance on expression classification.Source: §6 · p. 10
- Despite domain shift, a linear layer on FAb-Net’s learned embedding can predict pose, landmarks, and expression on other datasets.Source: §6 · p. 10
- FAb-Net learns pose and expression by observing faces move and change across videos, without hand labels.Source: §6 · p. 10