Source-linked AI summary
Self-Supervised Audio-Visual Co-Segmentation
Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, Antonio Torralba
TL;DR
The Mix-and-Separate model required synchronized audio and video at inference, limiting use when such synchronized input is unavailable. This paper learns sparse semantic feature channels for independent image or audio inference after joint training, and reports promising segmentation and source-separation results on AVE.
Problem
The Mix-and-Separate model could only be applied to videos with synchronized audio after training, limiting use when synchronized test input is unavailable.
Source: §1 · p. 1, §2.2 · p. 1Method
A staged sigmoid-to-softmax schedule creates sparse feature channels that are assigned dataset categories, supporting independent image and audio inference.
Source: §1 · p. 1, §2.2 · p. 2Results
Takeaways & Limitations
The learned representations support image-only segmentation and audio-only source separation after audio-visual training.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this paper we develop a neural network model for visual object segmentation and sound source separation that learns from natural videos through self-supervision. The model is an extension of recently proposed work that maps image pixels to sounds. Here, we introduce a learning approach to disentangle concepts in the neural networks, and assign semantic categories to network feature channels to enable independent image segmentation and sound source separation after audio-visual training on videos. Our evaluations show that the disentangled model outperforms several baselines in semantic segmentation and sound source separation.
1. INTRODUCTION
Image segmentation and sound separation have traditionally relied on labeled data, while the Mix-and-Separate model required synchronized audio and video at inference. This work disentangles network concepts to enable independent image and audio inference, with promising results on both tasks.
1. INTRODUCTION
- The work develops audio-visual co-segmentation using natural video correspondence, in a field where traditional approaches relied on large labeled datasets.Prior work used one modality to supervise the other or jointly used sound and vision; this paper develops a co-segmentation model.Source: §1 · p. 1
- The Mix-and-Separate model could only be applied to videos with synchronized audio after training.That inference constraint limited use with test inputs containing only video frames or sound mixtures.Source: §1 · p. 1
- A staged sigmoid-to-softmax learning approach produces sparse, disentangled feature channels that can be assigned to semantic categories.The category-to-channel assignments support independent inference and model interpretability.Source: §1 · p. 1
- The model achieves promising semantic segmentation and sound-source separation on a dataset of over 4,000 naturally occurring audio-visual videos.The authors evaluate image-only and audio-only tasks, which were not possible with the previous model.Source: §1 · p. 1
2. APPROACH
The approach trains audio and image networks jointly on synthetic mixtures, then learns sparse representations and matches dataset categories to feature channels. Those assignments support independent image segmentation and audio-source separation after training.
2.1. Self-supervised Cross-modal Training
- Two videos form a synthetic mixture; the image network extracts features from one video, and the audio network and synthesizer predict a mask for its sound.The network is trained through self-supervision without labeled data.Source: §2.1 · p. 1
2.2. Disentangling Internal Representations
- A sigmoid-to-softmax schedule with decreasing temperature encourages sparse, disentangled feature representations before audio-visual fusion.The schedule starts with diverse sigmoid activations, then moves toward one-hot outputs; the technique does not sample from the Gumbel distribution [21].Source: §2.2 · p. 2
2.3. Category to Channel Assignment for Co-segmentation
- After training, category labels are matched to feature channels so the image and audio networks can select category-specific outputs without the synthesizer.Image segmentation selects and thresholds an assigned channel’s activation map; audio separation applies assigned channels as masks to the mixture spectrogram. Table 1 covers classification and sparsity, while Table 2 covers sound separation and semantic segmentation.Source: §2.3 · p. 2, Table 1 · p. 3, Table 2 · p. 3
3. EXPERIMENTAL SETUP
The system uses image and audio analysis networks trained and evaluated on the AVE dataset. Its activation schedule progresses from sigmoid training to softmax fine-tuning, with sparsity measured on the visual feature vector.
3.1. Models
- The image network is a dilated ResNet-18, and the audio network is a modified U-Net; an audio synthesizer combines their features during training.For 224 × 224 frames, the image network produces K 14 × 14 activation maps; the audio network produces K 256 × 256 spectrogram feature maps.Source: §3.1 · p. 3
3.2. Dataset
- The AVE dataset contains 4,143 videos across 28 event categories, divided into training, validation, and test sets.Videos were cropped into six-second clips, frames were sampled at two per second, and audio was converted to 256 × 256 log-frequency spectrograms.Source: §3.2 · p. 3
- Figure 2 shows that the best-performing model’s feature-channel activations correspond to categories.Source: Figure 2 · p. 3
3.3. Activation Learning Schedule and Sparsity
- Training uses a fixed sigmoid activation, followed by softmax fine-tuning with custom temperature schedules; sparsity is measured on the post-activation visual feature vector.Fine-tuning also reduces learning rates by a factor of five; the schedules vary initial temperature, fine-tuning epochs, decay rate, and decay epochs.Source: §3.3 · p. 3
4. EXPERIMENTAL RESULTS
The experiments assess disentanglement and classification, audio-only source separation, and image-only semantic segmentation. The reported results favor different model settings across tasks, with segmentation masks often having imperfect boundaries.
4.1. Disentanglement and Classification
- The experiments used 32 channels, which the authors found effective and efficient for matching 28 AVE categories while leaving extra channels for silence or noise.Hyperparameters were selected by random search.Source: §4.1 · p. 4
- 45.9% classification accuracy for “Sigmoid & Softmax E” exceeded the model’s baseline variants, while the supervised ResNet-18 linear SVM reached 68.9%.The supervised baseline used ImageNet-pretrained ResNet-18 features; the proposed model’s smaller feature vector enables selection of discrete sources.Source: §4.1 · p. 4
- The best model’s visual channels generally responded to one or a few input categories, with some shared responses among semantically related categories.The authors attribute most misclassifications to relatively fine-grained confusions.Source: §4.1 · p. 4
- Softmax fine-tuning was necessary for sparse activations, and a gradual temperature decay from 1 to about 0.01 gave the best schedule.Ending at a temperature that was too high or too low led to suboptimal performance.Source: §4.1 · p. 4
4.3. Semantic Segmentation
- “Sigmoid & Softmax E” achieved the highest SDR and SIR, while “Sigmoid & Softmax C” achieved the best IoU and “E” performed nearly as well.The source-separation results are shown in Fig. 3; segmentation boundaries were often imperfect, possibly because of low-resolution activation maps and weak supervision.Source: §4.3 · p. 4, Figure 4 · p. 4, §4.2 · p. 4
5. CONCLUSION
The paper presents self-supervised audio-visual training with disentangled feature representations and category-to-channel assignments for independent image segmentation and sound-source separation.
5. CONCLUSION
- On AVE, the approach reports promising semantic segmentation and source separation after learning category-aligned, disentangled features from videos through self-supervision.The conclusion describes the approach as training networks for source separation on a large video dataset, then assigning dataset categories to network feature channels.Source: §5 · p. 4