PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Sound-Guided Image Manipulation

Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Sang Ho Yoon, Jinkyu Kim, Sangpil Kim

arXiv:2208.14114v3cs.CV

TL;DR

Text-guided image manipulation can be limited in expressing rich, dynamic scene semantics, while prior sound-guided methods could alter visual details unrelated to the audio. The paper learns a CLIP-based joint image-text-sound representation and uses latent optimization to guide manipulation with audio. The authors report robust, more diverse sound-guided results, including visual changes proportional to audio intensity.

  • Problem

    Text can be limited in conveying dynamic and nuanced scene semantics, and prior sound-guided manipulation could change image attributes unrelated to its audio input.

    Source: §1 · pp. 2, 4

  • Method

    The method extends CLIP’s image-text space with audio representations learned through weakly paired audio-visual and self-supervised audio learning, then optimizes image latents against audio.

    Source: §1 · p. 4, §3.1.2 · p. 14, §3.2.1 · p. 17

  • Results

    The authors report robust target-attribute manipulation, more diverse and detailed intensity-related results than state-of-the-art text-based methods, and visual style changes proportional to audio intensity.

    Source: §1 · p. 7

  • Takeaways & Limitations

    The learned joint embedding supports sound-guided image manipulation that responds to audio semantics and intensity.

    Source: §1 · p. 7, §7 · p. 35

    The method requires a trained StyleGAN latent space, increasing learning cost in real applications and limiting the domain of visual content.

    Source: §6.1 · pp. 33–34

Abstract

from arXiv · show

Recent successes suggest that an image can be manipulated by a text prompt, e.g., a landscape scene on a sunny day is manipulated into the same scene on a rainy day driven by a text input "raining". These approaches often utilize a StyleCLIP-based image generator, which leverages multi-modal (text and image) embedding space. However, we observe that such text inputs are often bottlenecked in providing and synthesizing rich semantic cues, e.g., differentiating heavy rain from rain with thunderstorms. To address this issue, we advocate leveraging an additional modality, sound, which has notable advantages in image manipulation as it can convey more diverse semantic cues (vivid emotions or dynamic expressions of the natural world) than texts. In this paper, we propose a novel approach that first extends the image-text joint embedding space with sound and applies a direct latent optimization method to manipulate a given image based on audio input, e.g., the sound of rain. Our extensive experiments show that our sound-guided image manipulation approach produces semantically and visually more plausible manipulation results than the state-of-the-art text and sound-guided image manipulation methods, which are further confirmed by our human evaluations. Our downstream task evaluations also show that our learned image-text-sound joint embedding space effectively encodes sound inputs.

1. Introduction

Sound can express dynamic and nuanced scene semantics that text alone may not fully convey, but prior sound-guided manipulation could alter visual attributes unrelated to the audio. The paper addresses this by learning weak audio-visual pairings and optimizing image latents against audio, reporting robust manipulation and richer intensity-related results.

1. Introduction

  • Sound inputs provide continuity and dynamics over time, allowing different semantic or sentiment levels that discrete text alone may not fully express.Thunder clips, for example, can differ in loudness and sound characteristics.Source: §1 · p. 2
  • Prior sound-guided manipulation could change audio-unrelated attributes because training used only one strong audio-frame positive pair per video despite biases in audio-visual pairs.Examples include unintended changes to hair color, hairstyle, glasses, or beard.Source: Figure 2 · p. 3, §1 · p. 4, Figure 3 · p. 5
  • Weakly paired contrastive learning adds positive audio-image links across videos sharing a text description, using their shared visual features to reduce representation bias.The method minimizes KL divergence between visual-text and audio-visual cosine-similarity probability distributions.Source: §1 · p. 4
  • The method aligns audio representations with CLIP semantics, learns audio views to capture intensity, and optimizes a StyleGAN latent code to match input audio.The optimized latent code produces an image whose CLIP representation is closer to the audio input.Source: §1 · pp. 5–7, Figure 4 · p. 6
  • The authors report robust target-attribute manipulation, more diverse and detailed intensity-related results than state-of-the-art text-based methods, and effectiveness using audio features including temporal context, tone, and volume.They also report sound representation outperforming existing audio-visual joint representations for image manipulation.Source: §1 · p. 7
  • The paper extends its preliminary conference version with further investigations, including ablations, music style transfer, and discussion of limitations and future directions.Source: §1 · p. 7

2. Related Work

Related work spans text-guided image manipulation, sound-guided methods, StyleGAN latent-space control, and audio-visual representation learning. This paper extends CLIP-based image-text representations with sound and uses weakly paired audio-visual learning to address biases relevant to manipulation.

2.1. Text-Guided Image Manipulation

  • StyleCLIP and related methods use CLIP-based text-image embeddings and StyleGAN latent spaces to produce text-relevant image manipulations.The present work follows CLIP-space distance optimization while extending the embedding space to include sound.Source: §2.1 · pp. 7–8

2.2. Sound-Guided Image Manipulation

  • Earlier sound-guided work explored music-to-visual transfer or sound manipulation, but prior approaches could fail to retain unrelated visual cues, motivating weakly paired and intensity-aware learning.Music visualizers focused on reactions or limited semantic relationships, while the prior sound manipulation model could lose visual cues the sound should not change.Source: §2.2 · pp. 8–9

2.3. Interpreting Latent Space in StyleGAN

  • StyleGAN’s intermediate and extended latent spaces support interpretable image controls, whereas audio-reactive traversal based only on signal magnitude does not control sound meaning.The paper contrasts random latent traversal with guidance from a pre-trained sound representation.Source: §2.3 · p. 9

2.4. Audio-Visual Representation Learning

  • Audio-visual representation learning aligns modalities with contrastive learning, but this work adds text-query-sampled images as weak audio-visual pairs to reduce bias and improve manipulation realism.The paper notes that audio-visual training has fewer positive pairs than CLIP’s image-text training.Source: §2.4 · p. 10

3. Method

The method learns a CLIP-based joint representation for image, text, and sound, then uses direct latent-code optimization to manipulate an image in response to audio.

3. Method

  • The method extends CLIP’s pre-trained image-text embedding space with sound by training an audio encoder to align audio with the other modalities.It uses audio-text and audio-audio contrastive losses, plus a weakly paired audio-image contrastive loss.Source: §3 · p. 11
  • Weakly paired audio-image contrastive learning improves the quality of the joint representation without requiring it to be learned from scratch.The approach leverages the pre-trained CLIP visual-textual representation because multimodal datasets are limited.Source: §3 · p. 11
  • Given the joint embedding space, direct latent-code optimization manipulates an image according to an audio input, such as giggling or sobbing.The approach is similar in strategy to text-guided models such as StyleCLIP, but uses sound as the guidance modality.Source: §3 · p. 11

3.1. Multi-Modal Latent Representation Learning

The section builds a joint representation from audio, text, and image inputs, then adds audio-only intensity learning and weakly paired audio-image alignment. Contrastive objectives and visual-text similarity distributions guide how these representations relate.

3.1.1. Extending CLIP Embedding Space with Sound

  • Three encoders map same-video audio, text, and image inputs into d-dimensional representations; InfoNCE pulls positive audio-text and audio-visual pairs together and pushes negative pairs apart.The method forms minibatch similarity matrices for these modality pairs.Source: §3.1.1 · pp. 11–12
  • Text inputs are augmented through synonym replacement, word permutation, and random word insertion to improve representation quality and address limited audio-text data.Synonyms are selected from WordNet.Source: §3.1.1 · p. 13

3.1.2. Self-Supervised Learning towards Intensity-Aware Sound Representation

  • Audio self-supervision makes augmented views of the same input positive pairs and different audio samples negative pairs, aiming to capture subtle intensity differences.The paper motivates this objective with distinctions such as weak versus severe thunderstorms.Source: §3.1.2 · pp. 13–14

3.1.3. Weakly Paired Contrastive Learning for Audio-Visual Joint Embedding Space

  • The weakly paired objective matches audio-visual and visual-text similarity distributions with KL divergence rather than enforcing hard identity labels.The method minimizes divergence between the distributions within minibatches.Source: Figure 7 · p. 15, §3.1.3 · p. 16
  • Weakly paired learning treats images from other videos with the same text labels as related to the audio, extending positives beyond same-video pairs.New images are sampled using the audio's text labels, and CLIP transfers visual-text link information to the audio-visual similarity matrix.Source: §3.1.3 · pp. 15–16

3.2. Sound-Guided Image Manipulation

The method optimizes a StyleGAN latent code toward an audio-guided image while regularizing its distance from the source. Adaptive layer masking controls which style layers change, and an identity loss preserves facial identity in human-image manipulations.

3.2.1. Direct Latent Code Optimization

  • Audio-guided manipulation optimizes each image instance at inference by moving its source latent code toward an audio-driven code in the learned joint embedding space.The optimization balances audio alignment with similarity to the source image.Source: §3.2 · pp. 16–17, §3.2.1 · p. 17
  • The optimization regularizes the distance between the audio-driven and source latent codes, with higher regularization weights preserving more source-image content.The source code can be randomly generated or obtained from an input image through GAN inversion.Source: §3.2.1 · p. 17

3.2.2. Adaptive Layer Masking

  • Adaptive layer masking assigns different weights to StyleGAN layers because their latent codes encode different style attributes.A trainable layer mask is updated during latent optimization to adapt the manipulation to the semantic cue.Source: §3.2.2 · pp. 18–19

3.2.3. Identity Loss

  • ArcFace-based identity loss supports facial-expression changes while preserving personal identity, and is disabled for non-human image manipulations.The method sets λID to 0 for non-human images.Source: §3.2.3 · p. 19

4. Experiments

Across qualitative comparisons, the method produces plausible sound-guided manipulations, reflects sound intensity, and preserves image attributes unrelated to the audio. The experiments use pretrained encoders and generators with audio-text and image datasets, and include comparisons with text-guided methods, sound-based models, and alternative multimodal embedders.

4.1. Implementation and Training Details

  • The model uses pretrained CLIP image and text encoders, a trained Swin audio encoder, and StyleGAN2 for image manipulation.Audio is converted to Mel-spectrogram features; training uses 50 epochs and a batch size of 384.Source: §4.1 · p. 19

4.2. Datasets

  • AudioSet and VGG-Sound provide audio-video training data, while FFHQ, LSUN, WikiArt, and LHQ provide images for training the generator.The paper reports using 17,153 balanced and 1,617,939 unbalanced AudioSet clips, plus 182,342 VGG-Sound videos.Source: §4.2 · pp. 20–21

4.3.1. Comparison Against Text-Guided Manipulation

  • Sound-guided manipulation handles multiple labels together, such as “baby” and “crying,” whereas text-guided methods emphasize one or the other.The authors also report better manipulation quality than TediGAN and StyleCLIP.Source: Figure 9 · p. 20, §4.3.1 · p. 21

4.3.3. Comparison with Existing State-of-the-Art Sound-Based Style Transfer Models

  • The proposed method produces more visually plausible manipulations than prior sound-guided work, which often shifts colors such as skin tones.For fire-crackling and rain inputs, it produces images consistent with the sound semantics where the compared sound-based methods miss those cues.Source: §4.3.1 · p. 22, §4.3.3 · p. 22, Figure 10 · p. 23

4.3.2. Audio Intensity-Aware Image Manipulation

  • Sound intensity correlates with style variation, including changes for rain and thunder and for fire-crackling, gunfire, and bomb sounds.The reported examples include intensity or attribute variations not explicitly represented in the embedding-space pretraining descriptions.Source: §4.3.2 · p. 22

4.3.4. Comparison with Existing Multi-modal Embedders

  • The learned joint embedding space generally yields better manipulation results than Wav2CLIP, AudioCLIP, and Lee et al.’s embedder across the shown sound inputs.The figure compares bird song, gunfire, bombs, fire crackling, chainsaws, thunder, and waves.Source: §4.3.4 · p. 23, Figure 11 · p. 24

4.3.5. Effect of Audio-Visual Weakly Paired Contrastive Learning

  • Weakly paired audio-visual contrastive learning preserves audio-irrelevant attributes such as glasses, hairstyles, and beards while supporting target attributes together.The examples include manipulations satisfying “baby” and “laughing” simultaneously.Source: §4.3.5 · pp. 23–24, Figure 12 · p. 25

4.3.6. Effect of Adaptive Layer Masking

  • Adaptive layer masking makes manipulations semantically more reasonable by adjusting latent-code changes to the semantic cue.In the thunderstorm example, the result with masking shows rain and lightning, unlike the result without it.Source: §4.3.6 · pp. 24–25

4.4. Quantitative Analysis

Quantitative analyses report stronger audio representations with weakly paired learning and semantically rich sound-guided manipulations. Sound-guided latent codes move farther from the source than text-guided codes, producing more diverse and dramatic generated images.

4.4.1. Zero-Shot Transfer

  • Increasing positive audio-visual data with weakly paired contrastive learning improves sound representation over Lee et al. and yields an embedding space more suited to image manipulation.The paper compares representation quality with supervised and zero-shot audio classification approaches.Source: §4.4.1 · pp. 26–27

4.4.2. Semantic Accuracy of Manipulation

  • Audio-guided manipulation generally outperforms the text-driven approach on semantic-level classification of manipulated images.The evaluation uses a linear classifier on audio embeddings across the reported semantic labels.Source: §4.4.2 · p. 28

4.4.3. Distribution of Manipulation Direction

  • Sound-guided latent codes move farther from the source than text-guided codes, and their generated images are more diverse and dramatic.The comparison measures cosine similarity between source, audio-driven, and text-driven latent codes.Source: §4.4.3 · pp. 28–29, Table 2 · p. 28

4.5. Human Evaluations

In a study with 50 participants, the authors compared manipulated images from their method with StyleCLIP and Lee et al. Participants rated the method better on naturalness and gave its images an average perceptual-realism score of 4.333.

4.5. Human Evaluations

  • 58.4% of participants judged the method’s generated images better manipulated than those from the other approaches.The comparison included StyleCLIP and Lee et al.Source: §4.5 · p. 29
  • 4.333 was the method’s average perceptual-realism score on a Likert scale from 1 to 5.The human evaluation compared the method with StyleCLIP and Lee et al.Source: §4.5 · p. 29
  • Fifty Amazon Mechanical Turk participants evaluated 12 sound sources using naturalness and perceptual-realism questionnaires.Source images came from FFHQ, LSUN, and LHQ.Source: §4.5 · p. 29

4.6. Sound and Text-Guided Style Mixing

The section combines sound and text guidance through interpolation and layerwise StyleGAN style mixing. These operations support continuous transitions between modalities and allow their distinct attributes to be mixed.

4.6. Sound and Text-Guided Style Mixing

  • Interpolating text- and sound-guided latent vectors produces continuous intermediate meanings, with style changes controlled linearly.The interpolated vector is a weighted sum of the text-guided and sound-guided vectors.Source: §4.6 · pp. 29–30, Figure 15 · p. 30
  • Layerwise style mixing selects latent-code layers guided by audio and text to combine their respective image properties.The method uses sound-guided code in layers 1–9 and text-guided code in layers 10–18.Source: §4.6 · pp. 30–31
  • Sound guidance manipulates facial emotion, such as giggling, while text guidance controls background color.The example mixes dynamic sound characteristics with human properties conveyed by text.Source: §4.6 · p. 31

5. Applications

The applications extend sound-guided manipulation to animal faces, artistic paintings, and music style transfer. In the painting user study, participants more often preferred this method over the cited alternatives.

5.1. Sound-Guided Animal Face Image Manipulation

  • The method manipulates animal faces using sound inputs such as laughing, dog barking, and cat meowing.The experiments used a StyleGAN3 generator pretrained on AFHQ v2, which contains over 15k images across cat, dog, and wildlife species.Source: §5.1 · pp. 31–32

5.2. Sound-Guided Artistic Painting Manipulation

  • 61.2% of 50 participants preferred our artistic painting results, compared with 26.7% for Lee et al. and 12.1% for the text-based approach.The method was demonstrated on WikiArt paintings with audio cues including laughing, crying, bombs, rain, and birds.Source: §5.2 · p. 32, Figure 18 · p. 33

5.3. Music Style Transfer

  • For music style transfer, the method outperforms previous works and produces robust manipulation for music it encounters for the first time.The evaluation used well-known 10-second music segments, including film soundtracks, a title song, and game audio.Source: §5.3 · p. 33

6. Discussion

The method depends on a trained StyleGAN latent space, which raises learning costs and limits the visual-content domain. The authors also identify possible misuse and social-bias risks associated with CLIP-based manipulation.

6.1. Limitations

  • A trained StyleGAN latent space raises real-application learning costs and limits the domain of visual content.The authors identify direct image generation from sound representations as future work for open-domain manipulation.Source: §6.1 · pp. 33–34

6.2. Societal Impacts

  • CLIP's unfiltered internet image-text training data may contribute social biases in sound-guided face manipulations, including stereotyped associations with criminality or housework.The authors also warn that the method could be misused to create fake news or propaganda.Source: §6.2 · pp. 34–35

7. Conclusion

The paper presents audio-guided image manipulation using StyleGAN2 latent space and CLIP embedding space. Its approach aligns latent codes with audio, and the authors report responsive manipulations across audio inputs.

7. Conclusion

  • The method maps user-provided audio into StyleGAN2 latent space and CLIP embedding space for semantic-level image manipulation.The conclusion describes alignment of the latent code with audio to reflect audio context in the manipulation.Source: §7 · p. 35
  • Adding a semantic edge between audio and visual information through text yields a more robust sound representation for fine features.Source: §7 · p. 35
  • The authors report responsive image manipulations across varied audio inputs and identify multimodal embedding-space traversal as applicable to many multimodal contexts.Source: §7 · p. 35

Related papers

Loading 2208.14114v3…