PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Cross-Modal Audio-Visual Generation

Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, Chenliang Xu

arXiv:1704.08292v1cs.CVcs.MMcs.SD

TL;DR

The paper addresses the limited study of generating one audio-visual modality from the other, beyond multimodal retrieval. It uses conditional GANs, specialized encodings, and paired musical-performance datasets, and reports generation to a good extent across instrument- and pose-level tasks. The authors also identify low I2S accuracy and difficulty quantifying pose-generation quality as limitations.

  • Problem

    Cross-modal audio-visual generation has not been systematically studied, while prior multimodal work largely focuses on retrieving existing samples rather than generating novel ones.

  • Method

    The paper uses conditional GANs with CNN-based modality encoders and decoders, explores sound transformations, and trains on two paired image-sound datasets for instrument- and pose-oriented generation.

  • Results

    The experiments demonstrate generation of one modality from the other to a good extent at both instrument-level and pose-level tasks.

  • Takeaways & Limitations

    The proposed problem formulation, GAN strategies, and two datasets establish a basis for future research in cross-modal audio-visual generation.

  • Takeaways & Limitations

    The I2S model generates log-mel spectrograms with low accuracy, while pose-generation quality is difficult to quantify.

Abstract

from arXiv · show

Cross-modal audio-visual perception has been a long-lasting topic in psychology and neurology, and various studies have discovered strong correlations in human perception of auditory and visual stimuli. Despite works in computational multimodal modeling, the problem of cross-modal audio-visual generation has not been systematically studied in the literature. In this paper, we make the first attempt to solve this cross-modal generation problem leveraging the power of deep generative adversarial training. Specifically, we use conditional generative adversarial networks to achieve cross-modal audio-visual generation of musical performances. We explore different encoding methods for audio and visual signals, and work on two scenarios: instrument-oriented generation and pose-oriented generation. Being the first to explore this new problem, we compose two new datasets with pairs of images and sounds of musical performances of different instruments. Our experiments using both classification and human evaluations demonstrate that our model has the ability to generate one modality, i.e., audio/visual, from the other modality, i.e., visual/audio, to a good extent. Our experiments on various design choices along with the datasets will facilitate future research in this new problem space.

1 INTRODUCTION

The paper introduces cross-modal audio-visual generation as a problem distinct from retrieval and applies conditional GANs to generate musical-performance images from sounds and sounds from images. It explores encoding choices, two generation scenarios, and paired datasets, with experiments indicating generation works to a good extent.

  • Motivation: Cross-modal generation must create novel, meaningful samples across modality spaces rather than retrieve existing database items.This requires learning a complex generative function that maps between visual and audio modalities.
  • Approach: Conditional GANs generate images conditioned on sounds and sounds conditioned on images for intersensory musical-performance generation.The paper studies instrument-oriented and fine-grained pose-oriented image generation.
  • Encoding: CNN-based encoders and decoders process images and transformed sound representations instead of raw audio waveforms.The paper explores multiple sound transformations for cross-modal encoding.
  • Datasets: Sub-URMP contains 17,555 paired images and half-second sound clips from 107 performances spanning 13 instrument types, while INIS pairs five-instrument images with solo clips.
  • Evaluation: Classification and human evaluations show conditional GANs generate one modality from the other to a good extent at both instrument and pose levels.The experiments also compare alternative design choices.
  • Contributions: The paper presents cross-modal audio-visual generation, cross-modal GAN structures and training strategies, and two datasets intended to support future research.

2 RELATED WORK

Related work has largely addressed multimodal retrieval, representation learning, and adversarial generation as separate problems. This paper applies conditional adversarial training to cross-modal audio-visual generation involving instruments and human poses.

  • Cross-modal analysis: Prior cross-modal analysis work primarily indexes and retrieves multimodal samples rather than generating new ones.
  • Representation learning: Multimodal representation methods learn shared or joint representations across audio-visual, image-text, and other modality pairs.
  • Adversarial training: Adversarial training has been applied to segmentation, localization, image translation, and speech enhancement, whereas this work targets music instruments and human poses.
  • Generative models: GANs use a generator and discriminator trained adversarially so generated samples resemble a target distribution while the discriminator distinguishes genuine samples.The target distribution is pdata, and random noise is drawn from pz.
  • Conditional generation: Conditional GANs direct generation using a condition variable passed to both generator and discriminator networks.In this paper, the condition is used for cross-modal audio-visual generation.

3 CROSS-MODAL GENERATION MODEL

The model uses separate conditional GAN pipelines for sound-to-image and image-to-sound generation, with modality-specific encoders, generators, and discriminators. It explores audio representations and trains models for instrument-oriented and pose-oriented generation.

  • The model has separate Sound-to-Image and Image-to-Sound networks, each comprising an encoder, generator, and discriminator.
  • The S2I generator combines a compressed sound encoding with random noise to produce a 64x64x3 synthetic image.
  • The adversarial strategy distinguishes true pairs, generated pairs, and wrong image-sound pairs sampled from the training data.
  • Instrument-oriented S2I uses one model across instruments, whereas pose-oriented S2I trains one model per instrument and samples wrong images within the correct category.
  • The I2S model is trained across the dataset to generate sound magnitude spectrograms for different instruments from performance images.
  • LMS is selected for audio encoding because it shows clearer patterns and the highest classification accuracy among the tested representations.
  • The image encoder uses the penultimate feature vector of a CNN whose classifier exceeds 95 percent accuracy on the testing set.

4 DATASETS

The paper introduces Sub-URMP and INIS because no existing dataset directly supports paired cross-modal audio-visual generation. The datasets pair musical-performance images with short sound clips across multiple instrument categories.

  • The authors compose two novel datasets, Sub-URMP and INIS, for training and evaluating cross-modal audio-visual generation.
  • Sub-URMP contains 13 instrument categories from URMP videos, split into 80% training and 20% testing without video overlap.
  • Sub-URMP yields 17,555 paired sound-image examples after segmenting videos into 0.5-second chunks and filtering quiet clips.
  • INIS combines manually screened ImageNet images from five instrument categories with 7,200 sampled 0.5-second audio chunks from 77 solo performances.

5 EXPERIMENTS

Experiments evaluate several sound-to-image variants, pose-oriented generation, and image-to-sound generation using qualitative, human, and classifier-based measures. Results favor classification-based sound encoding for instrument-oriented S2I, show varied pose generation, and reveal limitations in generated sound and late-training image quality.

  • 5.1 Model Variations: The Sub-URMP and INIS datasets provide examples and image-sound pair distributions for evaluating cross-modal generation.Sub-URMP examples contain roughly six complete solo songs per category; INIS includes generated images from S2I-A.
  • 5.1 Model Variations: The experiments compare three sound-to-image variants: S2I-C, S2I-N without mismatch information, and S2I-A with autoencoder-based sound encoding.S2I-C is the main classification-based model.
  • 5.2 Evaluating Instrument-Oriented S2I Generation: S2I-C generates higher-quality images than S2I-N and S2I-A because explicit sound-instrument classification produces less ambiguous conditioning features.The classifier also supplies a direct measure of encoding discriminativeness, unlike the autoencoder loss alone.
  • 5.2 Evaluating Instrument-Oriented S2I Generation: More than half of S2I-C images were judged realistic, one third received score 3, and S2I-C achieved mean score 1.81 versus 2.59 for ground truth.Human evaluation used 10 image sets per instrument, with scores from 0 to 3; images were evaluated at 64x64.
  • 5.2 Evaluating Instrument-Oriented S2I Generation: S2I-C produced especially strong human ratings for cello, flute, and double-bass images, including average scores of 1.9, 2.1, and 2.02.For cello, 18 of 30 images received score 3 and 25 received at least 2; flute had 15 score-3 images and 24 scoring at least 2; double bass had 18 score-3 images and 21 scoring at least 2.
  • 5.2 Evaluating Instrument-Oriented S2I Generation: Classifier evaluation finds S2I-A and S2I-N substantially worse than S2I-C, while S2I-C accuracy rises rapidly through epoch 35 before falling sharply toward epoch 50.Training and testing accuracies follow nearly the same trend, and late recovery can reflect repeated near-identical images rather than correct conditioning.
  • 5.3 Pose-Oriented S2I Generation: Pose-oriented S2I generates varied poses, but its instrument-trained sound encoder is not trained to classify poses; the authors expect note-trained encoding to improve pose matching.Figure 10 includes viola, violin, and cello examples with different movement ranges and multiple-person poses.
  • 5.4 Evaluating I2S Generation: I2S evaluation uses generated sound spectrograms because Mel filtering loses high-frequency information during conversion back to waveforms.Generated LMS reaches 11.17% classification accuracy versus 80% for the sound classifier on real LMS; generated LMS has less high-frequency and more low-frequency energy than real LMS.

6 CONCLUSION

The paper introduces cross-modal audio-visual generation using conditional GANs and evaluates it with two newly composed datasets. Results support generation between audio and visual modalities at instrument and pose levels, while audio generation accuracy and pose-quality quantification remain limited.

  • The paper makes an initial attempt to use conditional GANs for intersensory audio-visual generation.
  • Two novel datasets, Sub-URMP and INIS, are composed to evaluate cross-modal generation models.
  • The model generates one modality from the other to a good extent at both instrument and pose levels.
  • The model can generate a cello player's pose given the note being played.
  • The I2S model's LMS accuracy is low, while the quality of generated poses is difficult to quantify.
  • The authors identify stronger autoencoding and expert evaluation of reconstructed audio as future directions.
Loading 1704.08292v1…