Source-linked AI summary
Learnable PINs: Cross-Modal Embeddings for Person Identity
Arsha Nagrani, Samuel Albanie, Andrew Zisserman
TL;DR
The paper addresses whether faces and voices can be aligned in an identity-sensitive embedding without identity labels. It learns this space from talking-face videos using cross-modal self-supervision and curriculum-based negative mining, then evaluates unseen-unheard retrieval and TV-drama character labelling. The reported results show successful cross-modal verification and retrieval, including performance above random and outperforming prior work across gallery sizes.
Problem
Face and voice biometrics are usually learned independently, leaving open whether they can support a shared identity representation for cross-modal retrieval.
Method
The method jointly trains face and voice subnetworks on positive pairs from the same speaking face-track and negatives from different tracks, using curriculum-based hard-negative mining.
Results
The model performs above random for unseen-unheard cross-modal verification and outperforms previous work for all tested gallery sizes in cross-modal retrieval.
Takeaways & Limitations
The joint embedding supports cross-modal retrieval for unseen and unheard identities and one-shot character retrieval in TV dramas.
Takeaways & Limitations
Hard-negative selection is vulnerable to outliers and false negatives, while noisy voice segments can make facial appearance scores more useful and motivate better fusion strategies.
Abstract
from arXiv · showhide
We propose and investigate an identity sensitive joint embedding of face and voice. Such an embedding enables cross-modal retrieval from voice to face and from face to voice. We make the following four contributions: first, we show that the embedding can be learnt from videos of talking faces, without requiring any identity labels, using a form of cross-modal self-supervision; second, we develop a curriculum learning schedule for hard negative mining targeted to this task, that is essential for learning to proceed successfully; third, we demonstrate and evaluate cross-modal retrieval for identities unseen and unheard during training over a number of scenarios and establish a benchmark for this novel task; finally, we show an application of using the joint embedding for automatically retrieving and labelling characters in TV dramas.
1 Introduction
The paper asks whether faces and voices can share an identity-sensitive representation learned from unlabelled talking-face videos. It proposes cross-modal self-supervision and evaluates retrieval for identities unseen and unheard during training.
- The paper learns a joint face-voice embedding from unlabelled speech videos without explicit identity labels.Faces and voice segments from the same talking face form positives, while samples from different videos form negatives.
- Cross-modal embeddings allow faces to retrieve voice segments and voices to retrieve still photos or speech-less video segments.
- Unseen-unheard identity retrieval exceeds chance behaviour and goes beyond correlations based only on gender and age.
- The paper develops curriculum-based hard-negative mining because scheduling negative difficulty is crucial for unsupervised learning.
- The learned embedding is evaluated through cross-modal verification, 1 in N retrieval, and one-shot character labelling in TV dramas.
2 Related Work
Prior work studies multimodal matching and shared representations, while this paper targets fine-grained identity-sensitive alignment between faces and voices. Its approach builds on audio-visual self-supervision but removes the need for identity labels during training.
- Cross-modal research has addressed generation, matching, and retrieval, but this work focuses on constructing a shared representation of audio and vision.
- Audio-visual joint embeddings can be learned through self-supervised correspondence or cross-modal distillation from a teacher modality to a student modality.
- Unlike prior work on instruments, singing, and tools, this task requires fine-grained discrimination between individual faces and individual voices.
- Identity-sensitive embeddings must be invariant to environmental noise and speaker variation in lexical content, emotion, and intonation.
- Earlier work used identity labels for face-voice matching, whereas this paper approaches the problem without identity supervision.
3 Learning Joint Embeddings
The method maps faces and voices into a shared space using two neural subnetworks trained with positive and negative pairs from speaking face-tracks. Pair construction avoids temporal alignment so the embedding captures identity rather than synchronous nuisance factors.
- Face and voice networks map inputs from separate modality spaces into nearby points in a shared coordinate space for matching identities.
- A two-stream architecture produces 256-D face and voice embeddings and trains their pairwise relations with contrastive loss.
- SyncNet extracts contiguous speaking face-tracks by estimating correlation between audio and mouth motion.
- Segments from the same face-track are labelled positive, while segments from different face-tracks are labelled negative.
- Positive face frames are sampled uniformly from the speaking face-track rather than temporally aligned with the voice segment.
4 The Importance of Curriculum-based Mining
Unsupervised contrastive learning makes negative selection difficult because the hardest examples may be outliers or false negatives. The paper therefore uses curriculum-based mining to control negative difficulty during training.
- As datasets grow, contrastive learning faces quadratically many pairs, and easy gender-based negatives may not teach other identity factors.
- Hard-negative selection is delicate when training from scratch because the hardest samples may be outliers or false negatives.
- Standard OHEM selects the hardest minibatch pairs, but this setting makes that choice vulnerable to outliers and mislabeled negatives.
- The curriculum method samples K face-tracks, forms positive pairs by uniform face and three-second audio sampling, and selects appropriately hard negatives within the minibatch.
- The mining procedure uses a threshold negative whose distance from the anchor exceeds the anchor-positive distance, following semi-hard negative mining.
5 Dataset
The model is trained on VoxCeleb, a large-scale collection of noisy, varied talking-face videos, while identity labels remain unused during training. A curriculum parameter gradually increases hard-negative difficulty during training.
- Dataset: VoxCeleb contains over 100,000 speaking face-tracks from more than 20,000 challenging YouTube videos.Audio includes background noise, laughter, and varying acoustics; faces vary in lighting, quality, and pose.
- Dataset: Identity labels are not used during training, and unseen-unheard test identities are disjoint from the training identities.
- Curriculum mining: The curriculum increases the hard-negative threshold parameter by 10% every two epochs, from 30% to 80%, then holds it constant.The schedule is guided by mean positive, negative, and active-pair distances in minibatches.
6 Experiments
The experiments compare randomly initialized and pretrained face and voice subnetworks, including a teacher-student variant, using a 256-dimensional face embedding and contrastive-loss margin selection.
- Initialisation: The study compares training from scratch, pretrained subnetworks, and a teacher-student architecture with a frozen pretrained face subnetwork.The pretrained formulation initializes both subnetworks from single-modality identification models.
- Architecture: The face subnetwork uses VGG-M with batch normalization and maps a cropped 224 × 224 RGB face image to a 256-D embedding.Training augments faces with horizontal flips, brightness jitter, and saturation jitter.
- Training: A contrastive-loss margin of 0.6 was found optimal among the tested values 0.2, 0.4, 0.6, and 0.8.
7 Evaluation
Evaluation tests cross-modal verification, demographic effects, bias, and forced matching on seen-heard and unseen-unheard identities. The learned embedding performs above random on unseen-unheard verification and outperforms SVHF-Net across gallery sizes.
- Cross-modal verification: The verification benchmark tests whether a face and speech segment belong to the same identity, using separate seen-heard and unseen-unheard protocols.The evaluation uses AUC and EER, with randomly sampled positive same-identity pairs and negative different-identity pairs.
- Cross-modal verification: The model trained from scratch performs significantly above random on unseen-unheard identities, supporting unsupervised learning of a joint face-voice embedding.Embedding visualisations show loose identity-based clusters, while pretrained subnetworks provide additional gains.
- Demographic effects: Gender is the most influential demographic factor in cross-modal biometrics.The analysis evaluates negative pairs while holding gender, nationality, or age constant, with AUC reported for seen-heard and unseen-unheard identities.
- Cross-modal retrieval: For N = 10, qualitative forced-matching examples include four successful predictions and one failure case for each formulation.The figure presents both voice-to-face and face-to-voice formulations.
- Cross-modal retrieval: The joint embedding outperforms SVHF-Net for every forced-matching gallery size tested from N = 2 to 10.The method compares query and gallery embeddings with Euclidean distance and selects the closest result.
8 One-Shot Learning for TV Show Character Retrieval
The paper applies its joint face-voice embedding to one-shot character retrieval in TV shows, where a single face query must handle substantial appearance variation. Direct cross-modal comparison simplifies using audio and provides a slight improvement over either modality alone.
- One-shot TV retrieval recognises all appearances of a character using only a single face image, despite variation in pose, illumination, size, expression, and occlusion.
- A joint embedding lets the query face compare directly with the audio track, avoiding complex fusion systems required when face and voice representations are unaligned.
- Retrieval is evaluated among 17 categories: 16 principal characters and one background-character class, using AUC where higher is better.
- The fused face-and-voice method provides a slight improvement over using face or speech alone.
- Fusion is useful when one modality is weak, such as small, dark, or extremely posed faces, while facial appearance can help when audio is corrupted by noise or crosstalk.
9 Conclusion
The paper concludes that face and voice can be jointly embedded without identity labels and used for cross-modal retrieval of identities unseen and unheard during training. It also identifies curriculum negative mining as important for training from scratch and demonstrates broader applications and extensions.
- The authors demonstrate cross-modal retrieval for identities unseen and unheard during training and apply the joint embedding to TV-character retrieval.
- Curriculum negative mining is compared with random, semi-hard, and fixed-difficulty strategies in cross-modal verification experiments.
- Random mining produces results similar to chance, indicating collapse of training when positive and negative distances become indistinguishable.
- Fixed difficulty values of 0.5 and 0.8 classify easy cross-gender pairs but perform near chance on same-gender negatives, at 51.2% and 50.4%.
- With pretrained subnetworks, semi-hard and curriculum mining perform similarly, 86.4 versus 87.0, suggesting curriculum mining mainly helps models trained from scratch avoid poor local minima.
- Single-modality pretraining with a frozen face teacher produces worse results than training from scratch, possibly because the student must predict face features unavailable from audio.
B.3 Training Loss
The paper uses contrastive loss for learning the joint face-voice embedding and notes that alternative loss choices remain possible extensions.
- The experiments use contrastive loss for learning the joint face-voice embedding.
- A modified binary softmax loss slightly worsens performance relative to contrastive loss, with AUC values of 72.1% and 73.8%.
- Triplet loss is left as future work because sampling triplets across two modalities can introduce asymmetries.
C Evaluation Protocol for Cross-Modal Verification
The evaluation distinguishes identities seen during training from identities that are both unseen and unheard, using disjoint test identities and randomly selected positive or negative face matches for audio samples.
- The test protocol separates seen-heard and unseen-unheard identities, with the latter using disjoint identities and more samples.
- For each unseen-unheard audio sample, a positive same-identity or negative different-identity face image is selected at random from the test identities.
D Obtaining Age labels for the VoxCeleb dataset
Age labels for VoxCeleb are estimated automatically at the frame level because a person’s age may vary across videos. The resulting facetracks are organized into five age groups and summarized in a table and distribution figure.
- Label extraction: Age labels are obtained automatically for each detected face frame using a visual age classifier.Frames are sampled at 1fps, and the method replaces the original classifier’s pretrained VGG network with a Wide Residual Network.
- Label extraction: Age estimation is performed per frame because age can differ between videos of the same identity.VoxCeleb contains YouTube videos uploaded at different times, making identity-only age labels unsuitable.
- Age-group organization: The age distribution in the VoxCeleb test set is visualized using automatically obtained frame-level age labels.Figure 7 presents the distribution of estimated ages in the test set.