Source-linked AI summary
Objects that Sound
Relja Arandjelović, Andrew Zisserman
TL;DR
The paper addresses how audio and visual inputs can support cross-modal retrieval and how sounding objects can be localized without labelled data. It trains purpose-built networks on unlabelled video using audio-visual correspondence, achieving retrieval across modalities and semantic sounding-object localization. The authors also show that careful negative-pair sampling is important to avoid undesirable shortcuts.
Problem
Audio provides less direct semantic information than text, motivating methods that learn correspondences between images and audio from video.
Method
The paper trains retrieval and localization networks from scratch using audio-visual correspondence on synchronized, unlabelled video frames and audio.
Results
The AVE-Net supports cross-modal and within-modal retrieval, while the AVOL-Net localizes sounding objects without motion or flow information.
Takeaways & Limitations
Audio-visual correspondence can learn cross-modal retrieval and semantic sounding-object localization with appropriate network design.
Takeaways & Limitations
Motion-aware inputs can exploit lower-level correlations with sound, providing less incentive to learn semantic embeddings; the authors therefore use single-frame input for other experiments.
Abstract
from arXiv · showhide
In this paper our objectives are, first, networks that can embed audio and visual inputs into a common space that is suitable for cross-modal retrieval; and second, a network that can localize the object that sounds in an image, given the audio signal. We achieve both these objectives by training from unlabelled video using only audio-visual correspondence (AVC) as the objective function. This is a form of cross-modal self-supervision from video. To this end, we design new network architectures that can be trained for cross-modal retrieval and localizing the sound source in an image, by using the AVC task. We make the following contributions: (i) show that audio and visual embeddings can be learnt that enable both within-mode (e.g. audio-to-audio) and between-mode retrieval; (ii) explore various architectures for the AVC task, including those for the visual stream that ingest a single image, or multiple images, or a single image and multi-frame optical flow; (iii) show that the semantic object that sounds within an image can be localized (using only the sound, no motion or flow information); and (iv) give a cautionary tale on how to avoid undesirable shortcuts in the data preparation.
1 Introduction
The paper learns cross-modal audio-visual representations and sounding-object localization from unlabelled video using audio-visual correspondence (AVC). Its designs support retrieval across modalities while avoiding reliance on motion information.
- 1 Introduction: The motivating task is to localize, from an image and sound clip, the object that makes the sound without labelled examples.Figure 1 presents this sounding-object localization problem.
- 1 Introduction: Unlabelled videos provide synchronized frames and audio for self-supervised AVC training, distinguishing matching from mismatched pairs.Frames and audio from the same video and time form positives; mismatched videos form negatives.
- 1 Introduction: AVC encourages visual and audio representations that are both discriminative and semantically meaningful.The network must classify concepts in both modalities before judging whether they correspond.
- 1 Introduction: The visual network uses a single frame, preventing it from exploiting motion information to solve AVC.This encourages the visual representation to rely on appearance and semantic content rather than motion cues.
- 1 Introduction: The proposed networks enable cross-modal retrieval and localization of the image object producing a given sound.Both functionalities are trained from scratch without labels using the same AVC objective.
2 Dataset
The study uses a filtered subset of AudioSet videos as unlabelled training material. Video-level labels are retained only for quantitative evaluation, not for learning.
- 2 Dataset: AudioSet contains 10-second YouTube clips emphasizing audio events, with potentially multiple noisy video-level class labels.The labels are organized in an ontology.
- 2 Dataset: The dataset is filtered to 110 classes covering musical instruments, singing, and tools.Uninteresting categories such as silence, sine waves, and sound effects are removed.
- 2 Dataset: The videos are challenging because many have poor quality, invisible audio sources, or artificially inserted audio.These properties make audio-visual correspondence difficult to learn from the raw videos.
- 2 Dataset: No labels are used to train the methods; labels serve only to evaluate unsupervised retrieval quality.The dataset is treated as a collection of label-less videos during learning.
3 Cross-modal retrieval
The AVE-Net learns aligned audio and visual embeddings from unlabeled video using audio-visual correspondence, enabling cross-modal and intra-modal retrieval. It outperforms comparison methods, while careful sampling prevents shortcuts that inflate correspondence accuracy and harm retrieval.
- 3 Cross-modal retrieval: AVE-Net processes an image and one second of audio, then aligns their normalized embeddings using Euclidean distance for correspondence prediction.The architecture produces 128-D L2-normalized embeddings and uses their distance as the correspondence signal.
- 3 Cross-modal retrieval: Unlike L3-Net’s late concatenation, AVE-Net directly optimizes modality-specific features for retrieval alignment.L3-Net computes correspondence after fully connected layers, whereas AVE-Net makes the embeddings distance-aware.
- 3 Cross-modal retrieval: 81.9% AVC accuracy slightly exceeds L3-Net’s 80.8%, but retrieval is the primary test of the learned embeddings.The AVC objective is used as a proxy for learning useful representations.
- 3 Cross-modal retrieval: AVE-Net beats all baselines for cross-modal retrieval and also outperforms them for intra-modal retrieval, despite training only on cross-modal pairs.Intra-modal retrieval is attributed to transitivity through shared audio-visual feature space.
- 3 Cross-modal retrieval: Retrieval generally returns relevant items, while realistic errors include confusing zither sounds with acoustic-guitar sounds.The qualitative examples use aligned video frames to visualize audio retrieval results.
- 3.2 Extending the AVE-Net to multiple frames: Motion-based inputs performed underwhelmingly, so subsequent experiments use a single frame because motion could provide shortcuts rather than semantic embeddings.AVE+MF uses 25 frames, while AVE+OF combines one frame with ten optical-flow frames.
- 3.3 Preventing shortcuts and Implementation: Without matching the temporal sampling of negative audio pairs, AVC accuracy rises artificially to 87.6% versus 81.9%, while retrieval becomes 1–2% worse.Negative audio is therefore sampled only at the same 0.04-second multiples used for positive-pair alignment.
4 Localizing objects that sound
AVOL-Net learns to localize sounding objects from audio-visual correspondence without location or identity labels, using spatial visual descriptors matched against one audio embedding. It detects sound sources across varied scenes and audio contexts, though failures and ambiguity remain.
- Architecture: The MIL formulation treats local image regions as instances, so relevant regions correlate with the audio while background regions remain weakly correlated.The audio representation acts as a filter over image patches, similar to an attention mechanism.
- Architecture: AVOL-Net uses multiple spatial visual descriptors instead of global pooling, comparing each 14 × 14 descriptor with one audio descriptor to produce a localization map.The visual network replaces fully connected layers with 1 × 1 convolutions, removes feature normalization, and uses maximum similarity for the AVC correspondence score.
- Evaluation and results: AVOL-Net matches AVE-Net’s AVC accuracy, indicating that the localization-oriented MIL design does not reduce correspondence performance or semantic concept detection.This comparison is reported as equal accuracy rather than as a separate localization metric.
- Evaluation and results: Failures include noise from videos with overlaid music, attention to salient scene parts, and missed sounding objects; detections may also cover only discriminative object parts.The paper notes unresolved ambiguity about whether the sounding entity is an instrument, a component, a performer, or an entire ensemble.
- Evaluation and results: Mismatched-audio tests show that localization changes with the sound, supporting detection of the sounding object rather than only salient objects in the frame.The network can maintain separate object embeddings and highlight different objects when the audio context changes.
- Evaluation and results: 81.7% versus 57.2%, AVOL-Net outperforms center prediction on annotated instrument localization, showing it is not merely highlighting central salient objects.The evaluation samples 500 validation clips and compares the heatmap mode with a baseline that always predicts the image center.
- Evaluation and results: On independent video frames, the method detects sounding objects across poses and shots and can switch between interleaved speakers and instruments.No motion information or temporal smoothing is used, but the method can still follow varying audio contexts.
5 Conclusions and future work
The paper concludes that unsupervised audio-visual correspondence, combined with suitable network design, can support cross-modal retrieval and semantic localization of sounding objects. It identifies explicit soft attention as a possible improvement to AVOL-Net.
- Conclusions: Unsupervised AVC enables cross-modal retrieval and semantic localization of objects that sound when paired with appropriate network architectures.AVE-Net supports retrieval, while AVOL-Net supports object localization.
- Conclusions: The paper reports that AVE-Net outperforms supervised retrieval baselines, while AVOL-Net shows strong object-localization capabilities.These are the paper’s summarized outcomes for its two proposed functionalities.
- Future work: A proposed future improvement is replacing AVOL-Net’s current max-pooling with an explicit soft-attention mechanism.This is presented as a potential architectural modification rather than an evaluated result.
A AudioSet-Instruments
The AudioSet-Instruments subset provides the evaluation material for the paper’s unsupervised audio-visual experiments, while training uses videos without labels. It is drawn from AudioSet’s ontology-organized video-level labels.
- Dataset: The subset is derived from publicly available AudioSet, whose videos have potentially multiple ontology-organized audio class labels used only for evaluation.The methods treat the dataset as label-less during training.
A.1 Classes
The dataset is filtered to videos containing musical instruments, singing, or tools, producing 110 audio classes for the experiments.
- Class list: The listed classes include instruments, singing categories, bells, drums, guitars, keyboards, and tools such as drills.The passage provides the class enumeration; these examples summarize its categories.
A.2 Relevance
Retrieval relevance is defined using the AudioSet ontology, accounting for semantic distances between classes and multiple labels per video.
- Retrieval evaluation incorporates AudioSet ontology relationships rather than treating all class matches as equally relevant.For example, electric guitar should rank above drums for an acoustic-guitar query.
- AudioSet relevance uses ontology tree distance, with class relevance defined as r = C − d and C = 20.Acoustic guitar, electric guitar, and drums have tree distances 0, 2, and 5, respectively.
- Video-to-video relevance is the maximum relevance across all class pairs in their label sets.This handles videos with multiple labels when individual clips may not contain every video-level class.
B Initialization for the AVE-Net
AVE-Net initialization addresses the sign ambiguity of distance-based correspondence scoring, while AVE+OF uses separate image and flow processing before fusion.
- The AVE-Net’s distance-based scoring can learn anti-aligned embeddings unless the fc3 weights enforce the intended sign.The desired behavior is for smaller visual-audio distances to indicate greater similarity.
- The initialization strategy ensures that distance-based correspondence scoring has the intended similarity direction.The paper reports that simply initializing the tiny fc3 layer appropriately was sufficient.
- AVE+OF processes the RGB image and optical-flow streams independently through three conv-conv-pool blocks each before concatenating their outputs.The fused representation then passes through another conv-conv block.
C AVE+OF architecture
AVE+OF extends AVE-Net with optical flow while retaining its core frame-and-audio architecture and replacing only the vision subnetwork.
- AVE+OF ingests one RGB frame, 1 second of audio, and 10 optical-flow frames computed with TV-L1.Its vision subnetwork replaces AVE-Net’s single-frame vision input.
- The AVE+OF vision branch uses a two-stream design that processes image and flow independently before channel-wise concatenation.Each stream has three conv-conv-pool blocks, followed by an additional conv-conv block after fusion.
- The modified vision subnetwork is readily pluggable into the AVE-Net architecture.The rest of the AVE-Net structure remains shared with the original design.
D Additional AVE-Net results
Additional experiments report that AVE-Net outperforms the baselines across retrieval settings, including varying cutoff values and embedding dimensionalities.
- Figures 9 and 10 provide additional retrieval evidence supporting AVE-Net’s superiority over the baselines.These experiments complement the main retrieval results in Section 3.1.
- AVE-Net beats all baselines across embedding dimensionalities using average nDCG@30.The result holds regardless of the baseline embedding size.
- AVE-Net beats all baselines for every evaluated K on AudioSet-Instruments using average nDCG@K.The comparison is conducted on the AudioSet-Instruments test set.