PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Voices and Hearing Faces: Cross-modal biometric matching

Arsha Nagrani, Samuel Albanie, Andrew Zisserman

arXiv:1804.00326v2cs.CV

TL;DR

The paper asks how much identity-relevant information can be inferred across voice and face modalities without synchronized recordings. It develops CNN-based matching systems and evaluates static, dynamic, multi-way, and human-comparison settings. The models solve matching above chance, with dynamic information improving performance and the model exceeding humans on challenging same-demographic faces.

  • Problem

    The paper investigates whether a person’s face can be matched to their voice, and vice versa, when the modalities are not acquired simultaneously.

  • Method

    It trains CNN architectures on overlapping VGGFace and VoxCeleb identities for static, dynamic, and N-way cross-modal matching, with human testing as a baseline.

  • Results

    The model solves static and dynamic matching above chance, improves with dynamic images, and exceeds human performance on challenging faces sharing gender, age, and nationality.

  • Takeaways & Limitations

    The experiments strongly suggest that faces and voices contain cross-modal biometric information, including cues that untrained humans may find difficult to identify.

  • Takeaways & Limitations

    The architecture’s concatenation-based fusion cannot adjust the number of face streams during inference, motivating query pooling.

Abstract

from arXiv · show

We introduce a seemingly impossible task: given only an audio clip of someone speaking, decide which of two face images is the speaker. In this paper we study this, and a number of related cross-modal tasks, aimed at answering the question: how much can we infer from the voice about the face and vice versa? We study this task "in the wild", employing the datasets that are now publicly available for face recognition from static images (VGGFace) and speaker identification from audio (VoxCeleb). These provide training and testing scenarios for both static and dynamic testing of cross-modal matching. We make the following contributions: (i) we introduce CNN architectures for both binary and multi-way cross-modal face and audio matching, (ii) we compare dynamic testing (where video information is available, but the audio is not from the same video) with static testing (where only a single still image is available), and (iii) we use human testing as a baseline to calibrate the difficulty of the task. We show that a CNN can indeed be trained to solve this task in both the static and dynamic scenarios, and is even well above chance on 10-way classification of the face given the voice. The CNN matches human performance on easy examples (e.g. different gender across faces) but exceeds human performance on more challenging examples (e.g. faces with the same gender, age and nationality).

1. Introduction

The paper introduces cross-modal biometric matching: inferring a person’s face from their voice or their voice from their face without synchronized recordings. It uses deep learning and large-scale face and voice datasets to test static, dynamic, multi-way, and human comparison settings.

  • Cross-modal biometric matching asks whether a voice can identify the corresponding face, or a face can identify the corresponding voice.
  • Shared latent properties such as age, gender, ethnicity, and accent influence both facial appearance and voice.
  • The study uses CNNs trained on overlapping identities from VoxCeleb voices and VGGFace images.
  • The experiments compare still-image and dynamic-video matching, and extend binary forced choice to multi-way classification.
  • The model is compared with human performance, matching humans on easy examples and exceeding them on faces sharing gender, age, and nationality.

2. Related Work

Prior human studies generally found face–voice matching at chance with static faces and better performance when dynamic articulatory information was available. This paper instead targets large-scale, unconstrained matching in the wild.

  • Human Perception Studies: Human studies generally found static face–voice matching at chance, while dynamic visual information about articulatory patterns enabled matching.
  • Human Perception Studies: Prior work linked idiosyncratic speaking style, voice characteristics, and facial movement, suggesting dynamic information can support matching.
  • The paper differs from earlier small controlled studies by using large-scale data, unconstrained interview videos, and longer natural speech segments.

3. Cross-Modal Models

The paper models cross-modal matching with modality-specific face and voice streams followed by shared higher-level processing. It supports static, dynamic, and variable-cardinality N-way identification.

  • Binary V-F matching selects the positive face from a pair containing one face sharing the anchor voice’s identity and one negative face.
  • Training randomizes the positive face’s position and minimizes cross-entropy for the predicted label position.
  • Static Matching: The base model uses two parameter-sharing face streams and one voice stream, combining image inputs with voice spectrograms.
  • Shared higher layers capture cross-modal latent variables, while early layers specialize in modality-specific features.
  • Dynamic Matching: Dynamic matching adds dynamic-image streams for face inputs, using representations designed to capture temporal facial information.
  • N-way Classification: Query pooling enables the N-way architecture to handle an adjustable number of face inputs during inference.

4. Datasets and Training

The study combines overlapping large-scale face and voice datasets, constructs identity-disjoint evaluation splits, and trains networks end-to-end with modality-specific preprocessing and augmentation.

  • Datasets: The dataset uses 1,247 identities overlapping between VGGFace still images and VoxCeleb audio-visual videos.Training and testing identities do not overlap; validation uses names beginning with A or B, while testing uses C, D, and E.
  • Datasets: A challenging test set matches triplets by gender, broad age bracket, and nationality, using manually selected U.S. nationals aged 30–50.Nationality labels are used because they are typically more informative of accent than ethnicity.
  • Training: Each input triplet contains two face representations, either still images or videos, and one voice representation.The paper uses this triplet structure throughout the task formulation.
  • Training: All networks are trained end-to-end with stochastic gradient descent, batch normalization, momentum, weight decay, and a decaying learning rate.The stated minibatch size is 64, with momentum 0.9 and weight decay 5E−4.
  • Training: Face inputs use ImageNet-style cropping, flipping, and color-shift augmentation, while face and voice streams start from pretrained recognition models.The face and voice subnetworks are initialized from VGGFace and VoxCeleb models, respectively.

5. Experiments

Experiments evaluate static and dynamic cross-modal matching with controlled triplets, task-specific accuracy measures, and a human forced-matching benchmark.

  • Evaluation settings: Static evaluation presents two static face images and one speech segment, while dynamic evaluation uses two face-tracks and one audio segment.Training includes still images and frames extracted from VoxCeleb videos.
  • Evaluation settings: The protocol minimizes audio-visual synchrony so performance reflects identity-related cross-modal information rather than shared timing or lexical content.The stated concern includes mouth motion, sentence content, and emotional state when audio and video are extracted simultaneously.
  • Evaluation settings: At test time, the entire audio segment is processed with standard average pooling for all evaluation cases.This follows the procedure used in the VoxCeleb speaker-identification work.
  • Human benchmark: A human benchmark samples 500 static-test triplets and obtains judgments from 20 Amazon Mechanical Turk workers per sample.Workers completed the task in batches of five triplets.
  • Evaluation measures: Identification accuracy measures performance over the entire test set, whereas marginal accuracy measures speaker-specific performance.The dataset consists of labelled triplets containing one voice and two face candidates.

6. Results and Discussion

The model performs above chance across static, dynamic, and multi-way matching settings, with dynamic information improving results and difficult controlled comparisons exceeding human performance.

  • Matching results: Dynamic matching exceeds static matching by more than 3% for the V-F task.The paper attributes this increase possibly to visually dynamic information from articulatory patterns.
  • Matching results: Similar V-F and F-V accuracies suggest that cross-modal matching is highly symmetric.This agrees with the reported outcome of a prior human-perception study.
  • N-way classification: Top-1, top-2, and top-5 identification accuracy remain well above chance for every tested number of face images.Accuracy decreases as the number of test faces increases, while its relative improvement over chance remains relatively stable.
  • Human comparison: The model is comparable to humans on the static test set without GNA-variation removal but exceeds human performance when gender, nationality, and age variation is removed.The authors suggest subtle cross-modal biometrics may be difficult for untrained humans to identify, while noting possible dataset biases.
  • Speaker-specific results: The highest static V-F marginal accuracies occur for Celia Imrie, Cassandra Peterson, Cameron Boyce, and Estelle Harris.The paper notes that particularly distinctive identities can make the task easier, while accuracies remain above random across test identities.

7. Ablation Analysis

The ablation studies compare representations of facial dynamics, from static frames and single dynamic images to multiple dynamic images and RGB fusion. Multiple dynamic images improve accuracy, while dynamic images alone achieve 77%, indicating that motion carries cross-modal biometric information.

  • Dynamic representations: Single dynamic images summarize an entire face-track, whereas MDI computes k dynamic images from contiguous frame sets and fuses them through temporal pooling.The experiments use k = 10 and m = 10 for MDI.
  • Fusion: RGB and MDI inputs are combined in a fusion architecture and ensembled at test time.This formulation is identified as the best-performing dynamic setup in Table 3.
  • Dynamic representations: Multiple dynamic images capture local motion variations more effectively than a single dynamic image per face-track.The authors report a substantial accuracy increase with multiple dynamic images, possibly because they preserve local variations in motion.
  • Dynamic representations: Dynamic images encode lip motion while making latent variables such as age, gender, and ethnicity harder to discern visually.Figure 5 illustrates single-image and multiple-image representations, including frames with visible lip-motion information.
  • Dynamic representations: Dynamic-image-only inputs achieve 77% accuracy, suggesting that the network exploits dynamic cross-modal biometrics rather than relying solely on RGB appearance.The comparison is intended to test whether learned information comes from facial motion instead of RGB frames.

8. Conclusion

The paper introduces crossmodal face–voice matching and a CNN architecture for it. The model matches humans on easy binary forced-matching cases and exceeds them when candidate speakers share gender, age, and nationality.

  • The paper introduces crossmodal matching between faces and voices and proposes a corresponding CNN architecture.
  • Under binary forced matching, the model matches human performance on easy faces and exceeds human performance on challenging same-gender, same-age, same-nationality pairs.

A. Further Ablation Experiments

Further ablations examine audio duration and the effects of matching candidates on gender, age, and nationality. Performance is relatively robust for longer audio, while gender matching has the largest effect.

  • Audio duration: The study evaluates static matching with audio segments of 2/3, 1, 2, 3, and 4 seconds.Segments were extracted at both training and test time.
  • Audio duration: Convergence fails with 2/3-second segments, placing the minimum reliable duration between 2/3 and 1 second.
  • Audio duration: For longer durations, the method is relatively robust to variations in audio length.
  • Factor effects: 65.2% accuracy is obtained when triplets are matched on gender, compared with 75.0% for age, 78.2% for nationality, and 81.0% originally.
  • Factor effects: 63.9% accuracy occurs when gender, nationality, and age are all matched, and gender has the greatest effect on performance.The authors suggest this may reflect gender being more discriminative or limited age and nationality variation in the dataset.
  • Factor effects: The challenging GNA-var removed dataset contains 110 US speakers aged 30–50, with both genders represented.Nationality was obtained from Wikipedia and age estimates were obtained manually.

C. Amazon Mechanical Turk Study

The human baseline uses Amazon Mechanical Turk workers choosing the matching face for an audio segment. The protocol controls listening access, worker fatigue, repeated pair exposure, and low-quality responses.

  • Human evaluation: The benchmark uses audio segments restricted to at least 5 seconds, with densely sampled subsegments and averaged predictions for duration comparisons.
  • Human evaluation: The nationality distribution of speakers in the test set is summarized in Figure 6.
  • Human evaluation: Amazon Mechanical Turk workers choose which of two face images corresponds to a single audio segment.The selection buttons remain disabled until the audio has been played, and workers may replay samples.
  • Human evaluation: The test set contains 500 triplets shown to 20 workers in batches of five.Batches prevent the same speaker from appearing together, reducing opportunities to learn face–voice pairings.

D. Network Architecture Details

The architecture details include a voice-and-face network design, while worker-facing evaluation used an instruction-and-prediction webpage interface.

  • The filter and output sizes for both voice and face subnetworks are documented in Figure 8.
  • Workers received instructions and an illustrative example before submitting predictions through the evaluation interface.

E. Salient regions

The analysis examines which facial regions influence voice matching, while noting that voice spectrograms are harder to interpret visually. The model often uses localized lower-face regions but can selectively combine broader facial features, and it performs better on visually distinct face pairs.

  • The saliency visualizations were generated from samples in the GNA-var removed test set to inspect the model’s decision-making process.
  • Voice data is represented as spectrograms, which are less amenable to visual interpretation than facial inputs.
  • Average pooling in the voice subnetwork accommodates speech segments of variable length, with the apool6 dimension changing accordingly.For a three-second speech input, the caption reports N = 8 in apool6.
  • The model often emphasizes localized regions in the lower face, while sometimes using the nose, cheeks, eyes, or mouth across broader combinations.These patterns suggest selective use of multiple facial signals rather than one consistently decisive feature.
  • The model more readily distinguishes face pairs differing in gender and age on the static test set.
Loading 1804.00326v2…