PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Audio-Visual Learning: A Survey

Hao Zhu, Mandi Luo, Rui Wang, Aihua Zheng, Ran He

arXiv:2001.04758v1cs.CV

TL;DR

Audio-visual learning seeks to understand relationships between heterogeneous audio and visual modalities for challenging multimodal tasks. This paper surveys the field across four research areas, reviewing methods, datasets, and challenges. It concludes that mapping between modalities and enabling logical reasoning remain central open directions.

  • Problem

    Audio and visual data have heterogeneous representations, making it difficult to learn their relationships and limiting multimodal learning.

  • Method

    The paper provides a comprehensive survey organized into separation and localization, correspondence learning, generation, and representation learning.

  • Results

    The survey reviews state-of-the-art methods, commonly used datasets, and key challenges across four audio-visual learning areas.

  • Takeaways & Limitations

    Future progress requires better cross-modal mapping and logical reasoning, potentially using attention, memory banks, and knowledge graphs.

  • Takeaways & Limitations

    Audio-visual separation and localization studies often rely on unsupervised learning because training labels are scarce.

Abstract

from arXiv · show

Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities either to improve the performance of previously considered single-modality tasks or to address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods as well as the remaining challenges of each subfield are further discussed. Finally, we summarize the commonly used datasets and performance metrics.

1 Introduction

Audio-visual learning uses audio and visual modalities to improve existing single-modality tasks and address challenging multimodal problems. The survey organizes this research into four directions and reviews their methods, challenges, datasets, and metrics.

  • Audio-visual learning exploits relationships between audio and visual information to overcome limitations of single-modality perception tasks.
  • The survey divides audio-visual learning into separation and localization, correspondence learning, generation, and representation learning.
  • Audio-visual separation separates sounds from corresponding objects and localizes sounds within visual contexts, often improving separation in noisy scenes.
  • Correspondence learning studies global audio-visual relations through retrieval and audio-visual speech recognition.
  • Audio-visual generation synthesizes one modality from the other, whereas representation learning discovers useful representations directly from raw data.
  • The survey covers key methodologies, remaining challenges, commonly used datasets, and performance metrics across these research directions.

2 Audio-visual Separation and Localization

Audio-visual separation and localization separate sounds associated with speakers or objects and locate sound sources in visual contexts. The reviewed approaches progress from speaker-focused filtering to unsupervised, weakly supervised, and joint methods for more general settings.

  • Task scope: The field distinguishes speaker separation, which enhances a target voice, from object separation, which handles arbitrary sound-emitting objects.
  • Speaker separation: Visual information improves speaker separation by using facial movements, lip regions, landmarks, and multiple visual streams to filter noisy mixtures.
  • Speaker separation: Speaker-independent models can apply to any speaker after one training process and have outperformed speaker-dependent audio-visual separation methods.
  • Object separation and localization: Object-focused methods use motion, attention, temporal alignment, or learned audio-visual relations to localize and separate sources in complex scenes.
  • Unsupervised and weakly supervised learning: Because prior methods often require synchronized audio and video or specialized devices, later work explores video-only or sound-only inputs and unlabeled videos.
  • Simultaneous separation and localization: Mix-and-separate training combines audio from different videos, enabling models such as PixelPlayer to learn source separation without traditional supervision.

3 Audio-visual Correspondence Learning

Audio-visual correspondence learning studies global semantic relations between audio and visual modalities through matching, retrieval, and speech recognition. These tasks use cross-modal information to identify counterparts or improve recognition, while facing challenges involving representation, data requirements, and limited visual information.

  • Audio-visual correspondence learning covers matching and audio-visual speech recognition as tasks for discovering global semantic relations between modalities.
  • Audio-visual Matching: Cross-modality matching includes fine-grained voice-face matching and coarse-grained audio-image retrieval.Voice-face matching identifies corresponding identities, whereas audio-image retrieval searches for a relevant counterpart across modalities.
  • Voice-Facial Matching: Voice-face matching methods learn audio-visual embeddings, but early approaches handled fixed sample counts and did not fully exploit cross-modal correlation.Later methods used covariates or additional negative samples, but covariate-heavy approaches incurred high data costs and triplet loss was fragile with multiple samples.
  • Audio-image Retrieval: Audio-image retrieval maps modalities into a joint embedding space and compares counterparts using similarity-based objectives while attempting to preserve modality-specific characteristics.Pretrained networks, CNN replacements, intermodal ranking loss, and soft intramodal structure loss were used to reduce parameters and preserve modality-specific information.
  • Audio-visual Speech Recognition: Audio-visual speech recognition combines visual information with speech signals to provide semantic priors, such as scene context, for challenging recognition scenarios.Approaches evolved from two-step feature extraction and fusion toward neural encoders and end-to-end temporal models.
  • Audio-visual Speech Recognition: Lip-reading with the LipNet end-to-end architecture outperformed the earlier two-step strategy in experiments.LipNet used spatial-temporal convolutions, a recurrent network, and connectionist temporal classification loss for sentence-level sequence prediction.
  • Audio-visual Speech Recognition: Limited information in the visual modality may create a performance bottleneck, motivating models that combine audio and visual information across scenes, especially noisy conditions.Context-aware features, dual attention, and curriculum learning were among the strategies discussed for improving robustness and avoiding overfitting.

4 Audio and Visual Generation

Audio-visual generation studies synthesizing one modality from the other, extending generative modeling to cross-modal tasks. The survey reviews visual-to-audio and audio-to-visual generation across speech, scene sounds, images, body motion, and talking faces.

  • Audio-visual generation synthesizes audio from visual signals or visual content from audio, unlike tasks that use both modalities as inputs.
  • Visual to Audio: Visual-to-audio generation includes recovering speech from lip-area video and generating sounds associated with observed scenes.
  • Audio to Visual: Audio-to-visual generation includes producing images from audio, body motion, and talking faces.

Input lip sequence Output speech

Visual-to-audio generation reconstructs speech from lip video and synthesizes sounds from general videos, using learned relationships between visual signals and audio.

  • Visual-to-audio generation includes speech reconstruction from lip video and sound synthesis from general videos without scene limitations.
  • Lip Sequence to Speech: Lip-based methods predict spectral or learned audio features from silent video frames and reconstruct waveforms that produce understandable speech.Some approaches incorporate adjacent frames, optical flow, or recurrent networks for temporal information.
  • General Video to Audio: High-speed cameras can recover sounds from vibrations of suitable objects, linking visual motion with sound context.The approach applies to objects such as glasses of water, potted plants, or boxes of napkins.
  • General Video to Audio: Material- and interaction-conditioned models synthesize waveforms from videos of objects being hit or scratched, using recurrent feature extraction and instance-based synthesis.These models can identify sounds from different materials and learn that different actions produce different sounds.
  • General Video to Audio: Owens et al.'s laboratory-trained model was unsuitable for real-life applications, motivating an end-to-end video encoder and sound generator for in-the-wild videos.The later model directly predicts raw audio waveform samples and learns correlations across scenes and object interactions.
  • Audio to Visual: Audio-to-visual generation studies cover images, body motion, talking faces, and mutual cross-modality generation.The reviewed methods include conditional GANs, cyclic generative models, and audio encoders aligned with pretrained face representations.

Input audio Output image

Audio-to-image generation maps sound to visual content, with the reviewed material represented by a demonstration figure.

  • The section presents a demonstration of audio-to-image generation.

Input music

Audio-to-visual generation uses music or speech to synthesize body motion and talking faces, with methods increasingly modeling temporal, identity, and cross-modal relationships.

  • Music-conditioned generation includes body-motion and talking-face synthesis.
  • Moving Body Generation: Avatar animation methods generate motion using dimensionality reduction, hidden Markov models, Gaussian processes, and neural networks.
  • Moving Body Generation: Music-to-motion systems generate dance or performance videos using recurrent, autoregressive, and target-delay LSTM models.These methods produce choreography, body landmarks, or dynamics corresponding to musical input.
  • Moving Body Generation: Earlier motion-generation methods did not use intrinsic beat information, prompting models that learn acoustic-motion relations with LSTM autoencoders, masking, and temporal indexes.Other work uses weak motion-direction labels and conditional deep RNNs to generate long dance sequences.
  • Talking Face Generation: Talking-face generation has applications in movies, teleconferencing, talking agents, speech comprehension, and privacy-preserving speech enhancement.
  • Talking Face Generation: Methods synchronize generated faces with speech by predicting lip landmarks, using conditional GANs, optical flow, disentangled representations, and temporal-consistency losses.Later work targets arbitrary identities, mutual information, landmark and motion attention, and self-supervised facial-motion generation.

5 Audio-visual Representation Learning

Audio-visual representation learning seeks data-driven representations that capture semantic relationships between modalities, using synchronization, correspondence, clustering, and embedding tasks.

  • Representation learning is motivated because data-driven representations strongly affect machine-learning performance and real-world audio, video, and image features are difficult to specify algorithmically.
  • The reviewed methods span single-modality and dual-modality representation learning, with a basic pipeline summarized in Figure 8.
  • Synchronization and correspondence tasks exploit whether audio and video originate from the same video or are temporally aligned.
  • Single-Modality Representation Learning: A student-teacher process transfers visual identity knowledge to audio representations from unlabeled videos, but simultaneous audio-video representation learning remains unresolved.
  • Dual-Modality Representation Learning: Deep coclustering, K-means, and Gaussian semantic embeddings model cross-modal components while emphasizing semantic content over exact visual details.
  • Audio-Visual Correspondence: AVC trains visual and audio networks from matched and mismatched video pairs, while AVE-Net and 3D convolutions extend correspondence toward visual-area matching and sound localization.
  • Audio-Visual Temporal Synchronization: AVTS uses harder negative pairs consisting of unsynchronized audio and video, while earlier approaches assume the sound source appears in the video and only one source is present.
  • The section includes figures introducing AVC and AVTS tasks and summarizing recent audio-visual representation-learning studies.

6 Recent Public Audio-visual Datasets

The survey organizes public audio-visual datasets into speech-related and music or real-world event categories, spanning controlled laboratory recordings and in-the-wild Internet videos.

  • Speech-related datasets: Speech datasets capture faces with corresponding speech in laboratory-controlled or in-the-wild environments.Controlled datasets support tasks such as lip reading, talking-face generation, and speech reconstruction; in-the-wild datasets use interviews, talks, and movies.
  • Speech-related datasets: Lab-controlled speech datasets include GRID, TCD TIMIT, VidTIMIT, RAVDESS, Lombard Grid, OuluVS, and richly annotated SEWA.SEWA includes facial, linguistic, affective, gesture, interaction, and behavioral annotations.
  • Speech-related datasets: In-the-wild speech datasets address the limited real-world applicability of laboratory-trained models through interviews, talks, movies, and diverse online videos.Examples include LRW, VoxCeleb, AVA-ActiveSpeaker, and AVSpeech; LRW contains 500 sentences, while its variants contain 1000.
  • Music and event datasets: Music and real-world event datasets extend beyond facial videos to musical instruments, human actions, and other Internet-uploaded events.Examples range from ENST-Drums, C4S, URMP, and Sub-URMP to Kinetics, AudioSet, and YouTube-8M.
  • Music and event datasets: AudioSet contains 632 audio event classes and 2,084,320 human-labeled 10-second sound clips.Its coverage includes human and animal sounds, musical instruments and genres, and everyday environmental sounds.

7 Discussion

The discussion identifies modality heterogeneity, cross-modal mapping, reasoning, data scarcity, generation difficulty, and explainability as continuing challenges, while outlining research directions for addressing them.

  • Challenges: Audio-visual learning faces a fundamental gap between electrical audio signals and RGB visual representations, making their relationship difficult to model.The survey treats learning the relation between audio and vision as the basic challenge of the field.
  • Challenges: Separation and localization still struggle to distinguish object timbres and generate sounds for different objects.The discussion points to model designs such as attention mechanisms for handling varied objects.
  • Challenges: Cross-modality generation is difficult because it maps between probability distributions in a many-to-many setting and exposes artifacts noticeable to humans.The challenge combines heterogeneous modalities with sensitivity to differences between real and generated results.
  • Directions for future research: Future research should improve cross-modal mapping through attention and memory-bank ideas, while using prior information and knowledge graphs to support reasoning.The proposed directions simplify full machine logic by leveraging structured prior information in specific areas.
  • Directions for future research: Large, high-quality datasets and small-sample learning are important because dataset collection is labor- and time-intensive, while representation learning may mitigate data limitations.The discussion connects dataset development and general representation learning to broader AVL applicability.
  • Directions for future research: More complex networks often have unexplainable mechanisms, motivating research into robust and explainable audio-visual algorithms.The survey recommends learning from the essence of earlier explainable algorithms.

8 Conclusions

The paper reviews deep audio-visual learning as four research areas and summarizes datasets, challenges, and potential directions across the field.

  • Conclusion: The survey covers separation and localization, correspondence learning, audio and visual generation, and audio-visual representation learning.It also summarizes commonly used datasets and discusses challenges and research directions for each category.
Loading 2001.04758v1…