Sound-Guided Semantic Image Manipulation

• Text cues struggle to transfer specific, vivid sound semantics because discrete text cannot express sound’s continuous, dynamic variation.Sound-guided image generation also faces limited high-resolution audio-visual data and difficulty extracting temporal context, tone, and theme.Source: §1 · p. 2
• The proposed method aligns audio representations with textual and visual semantics, then directly optimizes a latent code in response to user-provided sound.The introduction illustrates this with an old car taking on a fire-truck-like appearance when given a siren sound.Source: §1 · p. 2
• The sound-based approach reflects audio information better and supports more diverse, detailed scene information than text-based image manipulation methods.The authors also report state-of-the-art performance on zero-shot audio classification.Source: §1 · p. 2
• The authors propose multi-modal contrastive losses, augmented-audio contrastive learning, semantic image manipulation from audio, and adaptive layer masking for more realistic outputs.Source: §1 · p. 2

Text-guided Image Manipulation

• Text-guided methods use GAN encoder-decoders or CLIP and StyleGAN priors to edit images from prompts, while TediGAN also uses GAN inversion and multi-modal mapping.Source: §2 · p. 2

• Figure 2 depicts triplet-based audio, text, and image alignment followed by direct optimization of a source latent code using user-provided audio.Source: Figure 2 · p. 3

Sound-guided Image Manipulation

• Earlier sound-guided work mainly visualizes music or maps sound magnitude to StyleGAN motion, rather than controlling latent navigation by sound meaning.The paper contrasts this with sound-semantic manipulation that can isolate an image region, such as changing facial emotion while preserving hair color.Source: §2 · pp. 2–3

Audio-visual Representation Learning

• Audio-visual representation learning maps modalities into a shared space and uses contrastive audio-visual pairs, but lacks data at CLIP’s scale.Source: §2 · p. 3
• The proposed audio encoder combines CLIP’s representational ability with self-supervised signals from audio data to learn an audio-specific representation.Source: §2 · p. 3

3.1. Multi-modal Latent Representation Learning

• Audio, text, and image encoders map inputs into matched d-dimensional vectors, anchoring audio learning to pre-trained CLIP because multimodal data are scarce.Source: §3 · p. 3, §3.1 · p. 4
• InfoNCE aligns positive audio-text pairs against negatives, while self-supervised learning on audio views and data augmentation improve audio representations.Audio inputs use SpecAugment; text inputs use synonym replacement, word permutation, and random insertion.Source: §3.1 · p. 4

3.2. Sound-guided Image Manipulation

• Directly optimizes StyleGAN latent codes toward audio embeddings, with adaptive layer masks and face-identity loss; shared audio-text embeddings also support style mixing.The identity loss is disabled for non-face manipulations.Source: §3.2 · pp. 4–5

4. Experiments

• The implementation uses a 512-dimensional audio representation from Mel-spectrogram features and StyleGAN2 for latent-code manipulation.The selected latent-code sizes are 18 × 512 for 1024 × 1024 images and 14 × 512 for 256 × 256 images.Source: §4 · p. 5

4.1. Qualitative Analysis

• For baby-crying examples, TediGAN emphasizes crying and StyleCLIP focuses on the baby, while our method handles both attributes simultaneously.Across the compared prompts, the authors also describe the audio-guided edits as natural style transfers that reflect multiple labels.Source: §4.1 · p. 6, Figure 5 · p. 6
• Changing thunder intensity or adding rain yields more diverse manipulation contexts than text guidance, illustrated by raining, weak thunder, and strong thunder inputs.The comparison is with StyleCLIP.Source: §4.1 · p. 6, Figure 6 · p. 6
• Joint audio–text style mixing assigns facial emotion such as giggling to sound and background color to text; adaptive masking makes the thunderstorm edit semantically reasonable, adding lightning and rain.For an 18 × 512 latent code, the described mixing uses sound-guided layers 1–9 and text-guided layers 10–18; the gate function rectifies the manipulation direction.Source: §4.1 · pp. 6–7, Figure 7 · p. 6, Figure 8 · p. 7

4.2. Quantitative Evaluation

• Our model outperforms previous studies in each zero-shot audio-classification task, and the comparison covers ESC-50 and UrbanSound8K classification accuracy and zero-shot inference.The authors attribute the richer audio representation to contrastive learning between audio samples; Table 1’s caption identifies its scope but supplies no cell values.Source: §4.2 · p. 7, Table 1 · p. 8
• Sound-guided latent codes generally move farther from the source than text-guided codes, and the resulting images are described as more diverse and dramatic.The comparison uses cosine similarity between source and manipulated latent codes.Source: §4.2 · pp. 7–8

4.3. User Study

• 100 participants evaluated the methods; 59.4% chose our generated image as best, and the authors report higher perceptual realism and naturalness than TediGAN and StyleCLIP.Naturalness was rated on a 1–5 Likert scale.Source: §4.3 · p. 8

5. Applications

• For artistic paintings, 73.3% of AMT responses favored audio guidance over text guidance for manipulation quality.The examples use a WikiArt-pretrained StyleGAN2; fire-crackling audio guides a painting toward a fire-crackling semantic cue.Source: §5 · p. 8
• For music style transfer, funny music produces a fairy-tale style and Latin music a red-color theme associated with passion.The authors describe the source latent code as close to the keywords of each music input.Source: §5 · p. 8

• User-provided audio is mapped into the StyleGAN2 latent space and CLIP embedding space.Source: §6 · p. 8
• Audio-aligned latent codes support responsive semantic edits for sounds including wind, fire, explosions, thunderstorms, rain, giggling, and nose blowing.Source: §6 · p. 9
• Drastic image-style changes can make identity preservation challenging in some cases.Source: §6 · p. 9
• Traversing the multi-modal embedding space can be used in applications with multi-modal contexts.Source: §6 · p. 9