Source-linked AI summary
Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
Yapeng Tian, Dingzeyu Li, Chenliang Xu
TL;DR
The paper asks whether videos can be parsed into temporally bounded events labeled as audible, visible, or both without dense annotations. It formulates the task as weakly-supervised MMIL, introduces temporal attention and pooling with bias and noise mitigation, and reports that parsing is tractable using video-level weak labels. The LLP dataset and experiments support leveraging multimodal temporal contexts while addressing modality bias and label noise.
Problem
Audio-visual video parsing requires recognizing event categories, temporal boundaries, and perceiving modalities, but dense modality, category, onset, and offset annotations are expensive.
Method
The paper formulates weakly-supervised parsing as MMIL and combines a hybrid attention network, attentive MMIL pooling, individual-guided learning, and label smoothing.
Results
Experiments on LLP show that audio-visual video parsing is tractable from video-level weak labels while leveraging multimodal temporal contexts and mitigating modality bias and label noise.
Takeaways & Limitations
Accurate audio-visual video parsing opens the door to potential applications including asynchronous audio-visual sound separation.
Takeaways & Limitations
The guided-loss and label-smoothing pipeline introduces new problems, motivating a future one-pass strategy such as modality dropout.
Abstract
from arXiv · showhide
In this paper, we introduce a new problem, named audio-visual video parsing, which aims to parse a video into temporal event segments and label them as either audible, visible, or both. Such a problem is essential for a complete understanding of the scene depicted inside a video. To facilitate exploration, we collect a Look, Listen, and Parse (LLP) dataset to investigate audio-visual video parsing in a weakly-supervised manner. This task can be naturally formulated as a Multimodal Multiple Instance Learning (MMIL) problem. Concretely, we propose a novel hybrid attention network to explore unimodal and cross-modal temporal contexts simultaneously. We develop an attentive MMIL pooling method to adaptively explore useful audio and visual content from different temporal extent and modalities. Furthermore, we discover and mitigate modality bias and noisy label issues with an individual-guided learning mechanism and label smoothing technique, respectively. Experimental results show that the challenging audio-visual video parsing can be achieved even with only video-level weak labels. Our proposed framework can effectively leverage unimodal and cross-modal temporal contexts and alleviate modality bias and noisy labels problems.
1 Introduction
The paper introduces audio-visual video parsing to identify event categories, temporal boundaries, and whether events are audible, visible, or both. It formulates weakly-supervised parsing as MMIL and proposes attention, pooling, guided learning, and label-smoothing techniques supported by the LLP dataset.
- Audio-visual video parsing identifies event categories, temporal boundaries, and whether events are audible, visible, or audio-visual.The task addresses events that may lack visual correspondences, such as out-of-view sounds.
- The paper formulates weakly-supervised parsing as a Multimodal Multiple Instance Learning problem and introduces a hybrid attention network for unimodal and cross-modal temporal contexts.The framework is designed to learn from video-level weak labels rather than dense temporal annotations.
- Attentive MMIL pooling adaptively aggregates useful audio and visual content across temporal extents and modalities.
- Individual-guided learning addresses modality bias, while label smoothing alleviates noisy labels.The methods target distinct issues arising when modalities do not consistently contain the same events.
- The LLP dataset contains 11,849 YouTube clips from 25 event categories, with sparse video-level labels for training and precise event annotations for evaluation.The authors report that audio-visual video parsing is tractable using only video-level weak labels.
2 Related Work
Prior work addresses temporal localization, sound event detection, and audio-visual learning, but audio-visual video parsing combines event localization with modality labeling for unconstrained videos.
- Temporal Action Localization: Temporal action localization typically targets actions using temporal candidates and full supervision, while weakly-supervised methods estimate action durations from video-level information.
- Temporal Action Localization: Unlike temporal action localization, audio-visual parsing includes motionless or out-of-screen sound sources and events perceived through either modality.
- Sound Event Detection: Weakly-supervised sound event detection focuses on audio events in constrained domains and generally ignores visual information.
- Audio-Visual Learning: Audio-visual parsing jointly models event categories, temporal boundaries, and perceiving modalities for unconstrained videos.
3 LLP: The Look, Listen and Parse Dataset
The LLP dataset was created for audio-visual video scene parsing because existing datasets were unsuitable. It provides weak video-level training labels and precise second-wise annotations for evaluation.
- The LLP dataset contains 11,849 YouTube clips spanning 25 categories and 32.9 hours of video.It includes events such as speaking, singing, crying, barking, instrument playing, vehicle sounds, and cleaning.
- Each 10-second video has at least one second of audio or visual events and video-level annotations indicating event presence or absence.These annotations facilitate weakly-supervised learning.
- The dataset includes 7,202 videos containing events from more than one category, with an average of 1.64 event categories per video.
- For evaluation, 1,849 randomly selected videos receive individual audio and visual event annotations with second-wise temporal boundaries.
4 Audio-Visual Video Parsing with Weak Labels
The task parses videos into temporally bounded audio, visual, and audio-visual events from paired snippets, while training uses only video-level labels. Its MMIL formulation separates audio and visual instances and must resolve temporal and modality uncertainty.
- Audio-visual video parsing groups video segments into semantically labeled audio, visual, and audio-visual temporal events.Because boundaries are annotated at second level, experiments parse scene-level rather than object- or instance-level events.
- Videos are divided into T non-overlapping one-second pairs of synchronized audio and visual snippets.
- An audio-visual event occurs only when audio and visual events coincide in time and belong to the same event category.
- Weakly-supervised training uses video-level labels, whereas inference predicts precise event label sets for every snippet.This makes the task a multimodal multiple instance learning problem.
- Audio and visual snippets at the same time are separate instances in MMIL, creating temporal and multimodal uncertainty during inference.A positive bag contains at least one positive snippet, with at least one modality containing an event.
5 Method
The method formulates weakly-supervised audio-visual video parsing as MMIL and combines hybrid temporal attention with attentive pooling to use multimodal context under video-level labels.
- The framework contains feature extraction, multimodal temporal modeling, and attentive MMIL pooling modules.It extracts snippet-level audio and visual features, models temporal context, and outputs video-level labels from instance-level predictions.
- The model uses video-level weak labels for all snippets and trains with binary cross-entropy after attentive pooling.The weak-label formulation avoids direct supervision for every audio and visual snippet.
- Hybrid Attention Network jointly models unimodal temporal recurrence, multimodal co-occurrence, and audio-visual asynchrony through self-attention and cross-attention.The network adaptively learns which bimodal and cross-modal snippets to use for each audio or visual snippet.
- Attentive MMIL pooling assigns tensorized temporal and audio-visual attention weights to select informative snippets and modalities for each event category.The pooling aggregates event predictions across temporal steps and audio or visual modalities to produce video-level probabilities.
- Audio and visual event probabilities are predicted independently with sigmoid outputs, while audio-visual event probabilities are derived from the individual modality predictions.Sigmoid supports multiple event labels per snippet rather than a single categorical choice.
- Individual-guided learning addresses modality bias, while label smoothing mitigates noisy modality-specific labels caused by events occurring in only one modality.A video-level label can contain an event absent from one modality, creating noise for that modality’s guided loss.
6 Experiments
Experiments evaluate audio-visual video parsing against weakly supervised unimodal and multimodal baselines and through ablations of the proposed components. The results show improved parsing across subtasks and metrics, with temporal and modality-aware learning addressing key failure modes.
- Experimental comparison: The evaluation compares against weakly supervised sound event detection, action localization, and audio-visual event localization methods on the LLP test dataset.All methods use the same audio and visual features for fair comparison.
- Experimental comparison: Our method outperforms the compared approaches on all audio-visual video parsing subtasks under both segment-level and event-level metrics.The comparison covers audio, visual, and audio-visual event parsing.
- Ablation studies: Individual-guided learning improves visual and overall audio-visual parsing by reducing the modality bias produced when training relies mainly on audio information.Training with both Lwsl and Lg performs better than training without individual guidance or with only Lg.
- Ablation studies: Attentive MMIL pooling assigns different weights to snippets and modalities, reducing incorrect labeling of audio-only or visual-only events as audio-visual events.Mean pooling equally aggregates instance scores and produces poorer individual audio and visual parsing.
- Ablation studies: HAN outperforms GRU, Transformer, and a non-temporal base model by jointly modeling unimodal recurrence, multimodal co-occurrence, and temporal asynchrony.The comparison indicates that temporal aggregation is important for audio-visual video parsing.
7 Limitation
The proposed guided-loss and label-smoothing steps mitigate modality bias and label noise but introduce new theoretical and procedural issues. The paper identifies a one-pass learning strategy as an open direction.
- Limitation: The guided loss can assign incorrect labels because some events occur in only one modality.This limitation arises from enforcing correct independent predictions for both modalities.
- Limitation: The staged mitigation methods introduce new problems, motivating a one-pass strategy such as modality dropout for addressing modality bias.Modality dropout is proposed as one possible future learning strategy.
8 Conclusion and Future Work
The paper concludes that weakly supervised audio-visual video parsing is tractable and that the proposed model leverages multimodal temporal contexts while addressing modality bias and label noise. Accurate parsing is positioned as a basis for sound separation and scene-aware video understanding applications.
- Conclusion: Experiments on the LLP dataset support learning audio-visual video parsing from cheap video-level weak labels.The conclusion describes the task as tractable under weak supervision.
- Conclusion: The proposed model leverages multimodal temporal contexts and mitigates modality bias and noisy labels.These findings summarize the reported capabilities of the framework.
- Future applications: Parsed scenes can provide cues for audio-visual scene-aware video dense captioning and question answering.These are presented as potential applications of audio-visual video parsing.
- Future applications: Asynchronous parsing can provide visual information for separating sounds whose sources are not temporally aligned with visible events.Figure 5 illustrates using asynchronous visual events to separate cello sound from an audio mixture.