Source-linked AI summary
Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
Soo-Whan Chung, Joon Son Chung, Hong-Goo Kang
TL;DR
Audio-visual synchronization can be framed as retrieving the most relevant audio segment for a short video clip. The paper learns cross-modal embeddings with multi-way matching, reporting stronger synchronization performance and visual-speech-recognition performance matching a fully supervised model with the same architecture.
Problem
Audio-visual synchronization is framed as retrieving the most relevant audio segment for a short video clip.
Source: §1 · p. 1Method
The method learns cross-modal embeddings by matching one visual feature against multiple audio candidates, using inverse distances with a softmax and cross-entropy loss.
Source: §1 · p. 1, §2.2 · p. 2Results
Across tasks, the method exceeds existing audio-visual synchronization methods and matches a fully supervised same-architecture model on visual speech recognition; for # frames = 5, synchronization performance rises from 75.8% to 89.5%.
Source: §4 · p. 4, §3.2 · p. 4, §3.1 · p. 4Takeaways & Limitations
The paper presents the multi-way embedding strategy as applicable to other cross-modal tasks.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper proposes a new strategy for learning powerful cross-modal embeddings for audio-to-video synchronization. Here, we set up the problem as one of cross-modal retrieval, where the objective is to find the most relevant audio segment given a short video clip. The method builds on the recent advances in learning representations from cross-modal self-supervision. The main contributions of this paper are as follows: (1) we propose a new learning strategy where the embeddings are learnt via a multi-way matching problem, as opposed to a binary classification (matching or non-matching) problem as proposed by recent papers; (2) we demonstrate that performance of this method far exceeds the existing baselines on the synchronization task; (3) we use the learnt embeddings for visual speech recognition in self-supervision, and show that the performance matches the representations learnt end-to-end in a fully-supervised manner.
1. INTRODUCTION
The introduction motivates self-supervised cross-modal learning as a way to use naturally co-occurring data, then proposes multi-way matching for audio-visual retrieval. It reports improved synchronization and visual speech recognition results for the learned embeddings.
1. INTRODUCTION
- Self-supervised learning can use Internet data without manual annotations, while cross-modal self-supervision draws supervision from naturally co-occurring streams.Examples of earlier representation-learning approaches include auto-encoders, inpainting, and predicting RGB images from grayscale images.Source: §1 · p. 1
- Prior audio-visual methods trained two-stream networks for binary matching or contrastive objectives, while SyncNet features were also used for lip reading and temporal alignment.The cited work leaves open whether binary matching objectives suit applications such as recognition and retrieval.Source: §1 · p. 1
- The proposed strategy learns cross-modal embeddings through multi-way matching, without explicit class labels and with a cross-entropy loss.The objective combines similarity-based methods with multi-class cross-entropy and is suited to retrieval across modalities.Source: §1 · p. 1
- The learned embeddings improve visual speech recognition over pairwise-objective representations, and the paper demonstrates them for audio-visual synchronization.Synchronization is framed as finding the most relevant audio segment for a short video clip.Source: §1 · p. 1
- Fig. 1 presents the trunk architecture for the audio and visual streams.Source: Figure 1 · p. 2
2. ARCHITECTURE AND TRAINING
The section describes audio and visual network trunks and contrasts pairwise correspondence baselines with the proposed multi-way matching strategy. The proposed objective compares one visual feature against multiple audio candidates and uses their distances in a cross-entropy loss.
2.1. Trunk architecture
- The audio trunk uses 13-dimensional MFCCs, while the visual trunk processes five stacked RGB face frames; both use VGG-M-based architectures.Audio inputs cover 20 frames; the visual stream uses a 5 × 7 × 7 first-layer filter to capture motion across frames.Source: §2.1 · p. 2
2. ARCHITECTURE AND TRAINING
- The baselines use pairwise correspondence objectives: SyncNet uses contrastive loss, and AVE-Net classifies correspondence from normalized embedding distance.The section compares the training strategy with AVE-Net and SyncNet, using the same input and layer configurations as SyncNet for the trunk architecture.Source: §2.2 · p. 2, §2 · p. 2, §2.1 · p. 2
2.2. Training strategies
- The proposed multi-way task compares one visual feature with multiple audio features, applies softmax to inverse Euclidean distances, and trains with cross-entropy.All N audio frames come from the same face track, but only one matches the video clip in time, encouraging learning of speech content rather than identity or utterance characteristics.Source: §2.2 · p. 2
3. EXPERIMENTS
The experiments evaluate the proposed matching strategy on audio-to-video synchronization and test whether its learned visual embeddings transfer to visual speech recognition.
3.1. Audio-to-video synchronisation
- Synchronization is framed as retrieval: select the audio segment closest to a video feature, averaging distances across multiple video samples when using more than five frames.The study varies N, trading the number of candidate audio features against the availability of video clips, and reports accuracy and clip counts by N.Source: §3.1 · p. 3
- For five visual frames, synchronization accuracy rises from 75.8% to 89.5% with the proposed method over the pairwise baseline.The reported synchronization results use the network trained with N = 40.Source: §3.1 · p. 4, Table 1 · p. 4
3.2. Visual speech recognition
- The visual speech evaluation uses LRW, a 500-word dataset with over 500,000 utterances and 25,000 reserved for testing.Source: §3.2 · p. 4
- The visual speech model uses the visual stream as its front end, followed by a two-layer temporal convolution back end and a 500-way softmax classifier.The model is named TC-5; the “5” denotes the feature extractor’s temporal receptive field.Source: §3.2 · p. 4, Figure 5 · p. 4
- On LRW word-level visual speech recognition, self-supervised features match the identical TC-5 network trained end-to-end with full supervision, without fine-tuning.The comparison is reported in Table 2.Source: §3.2 · p. 4, Table 2 · p. 4
4. CONCLUSION
The paper introduces a cross-modal matching and retrieval strategy that trains without explicit class labels. It reports superior synchronization performance and visual speech recognition performance matching a fully supervised model with the same architecture.
- The strategy trains cross-modal matching without explicit class labels while retaining favorable learning characteristics of cross-entropy loss.Source: §4 · p. 4
- The experiments report superior performance on audio-visual synchronization and visual speech recognition matching a fully supervised method with the same architecture.Source: §4 · p. 4
- The authors state that the embedding strategy should also apply to other cross-modal tasks.Source: §4 · p. 4