Positive Sample Propagation along the Audio-Visual Event Line

1. Introduction

• Large-scale multimodal pretraining datasets require substantial manual cleaning and annotation, motivating approaches that learn from noisy supervision or filter unpaired samples.The cited examples include audio-visual fusion and text-visual fusion.Source: §1 · p. 1
• AVE localization finds video segments containing an event that is both audible and visible, and assigns each event a category.The paper distinguishes this segment-level classification task from Multimedia Event Detection, which retrieves video clips associated with an event.Source: §1 · p. 1
• Prior AVE methods encode temporal or cross-modal relations, but can still fuse irrelevant audio-visual segment pairs that interfere with localization.The paper argues that aggregating only high-relevance pairs can improve AVE localization accuracy.Source: §1 · pp. 1–2
• PSP builds an all-pair audio-visual similarity map, removes entries below a threshold, and propagates features through the retained connections, including relevant unsynchronized pairs.The paper reports visualizations showing that PSP aggregates relevant features that are not necessarily synchronized.Source: §1 · p. 2
• The paper adds an audio-visual pair similarity loss for fully supervised learning and a temporal weighting branch for weakly supervised learning.The loss encourages correlated features for segments belonging to the same event; the branch assigns temporal weights to segment features.Source: §1 · p. 2
• The method achieves state-of-the-art performance on the standard AVE dataset in both fully and weakly supervised settings.The authors report that the proposed techniques consistently benefit the system and achieve this result when combined.Source: §1 · p. 2

2. Related work

• Audio-visual correspondence predicts whether an image and an audio signal correspond, motivating similarity-based evaluation of abundant audio-visual pairs in AVE localization.The cited correspondence methods assign higher similarity to corresponding pairs and lower similarity otherwise.Source: §2 · p. 2
• Sound-source localization targets relevant visual regions and has used methods that construct positive and negative audio-visual pairs, a related pairing problem to AVE localization.The cited approaches include triplet and contrastive losses.Source: §2 · p. 2
• Prior AVE approaches use concatenation, bilinear fusion, modality sentinels, transformers, or separate modality updates; PSP instead selects relevant audio-visual pairs for aggregation.The paper notes that the transformer approach divides frames into patches and uses adjacent frames, making it computationally intensive.Source: §2 · p. 2

3. Problem statement

• AVE localization determines whether each synchronized audio-visual segment depicts an event and identifies its category; videos are divided into continuous, non-overlapping, one-second segments.The visual and audio components are denoted separately for each segment.Source: §3 · p. 2
• In the fully supervised setting, every segment has a label indicating whether it is background or an event and, for events, the category.The sequence of segment labels forms the label for the entire video.Source: §3 · pp. 2–3
• The system flow encodes audio and video features, propagates features through threshold-selected positive pairs, and classifies events; its training adds a pair-similarity loss or weak-setting segment weights.The figure describes the similarity loss for supervised learning and an additional fully connected layer that weights segments in weak supervision.Source: Figure 2 · p. 3
• In the weakly supervised setting, the video-level label gives the proportion of audio-visual pairs containing an event, while each segment's event label is unknown.The passage describes this setting as more challenging than fully supervised localization.Source: §3 · p. 3

4.1. Overall pipeline

• The system extracts and encodes audio-visual features, applies PSP, then fuses the updated features for segment-level event and category prediction.Audio-guided visual attention focuses on visual regions related to audio, and separate Bi-LSTMs encode temporal relations.Source: §4.1 · p. 3, §4.2 · p. 3

4.2. Feature extraction and encoding

• Pretrained CNNs extract visual and audio features; audio-guided visual attention focuses visual features on audio-relevant regions before separate Bi-LSTMs encode temporal relations.Source: §4.2 · p. 3

4.3. Positive sample propagation (PSP)

• PSP constructs all audio-visual segment-pair connections, prunes negative and weak similarities by thresholding, then aggregates features through the retained positive connections.The resulting audio and visual features are updated using connection weights in an online feature aggregation process.Source: §4.3 · pp. 3–4

4.4. Classification

• The fused features are classified per segment in fully supervised learning; weakly supervised learning adds a branch that assigns temporal weights to segment features.The weakly supervised branch uses sigmoid-derived weights and averages predictions over time.Source: §4.4 · pp. 4–5

4.5. Objective function

• Fully supervised training combines category cross-entropy with a pair-similarity loss, while weakly supervised training uses binary cross-entropy.The pair-similarity loss is unavailable in weak supervision because segment-level event labels are unknown.Source: §4.5 · p. 5, §4.6 · p. 5

4.6. Discussion

• In weak supervision, average pooling gives temporal segments similar loss weights, making event-containing segments harder to focus on; the weighting branch aims to help distinguish and localize them.Source: §4.6 · p. 5

5.1. Experimental setup

• The AVE dataset contains 4,143 videos in 28 categories, partitioned into ten one-second segments; segment classification accuracy is the evaluation metric.Visual features use sampled frames and VGG-19, while audio features come from log-mel spectrograms processed by a VGG-like network.Source: §5.1 · p. 6

5.2. Quantitative Analysis

• Removing PSP lowers accuracy by 4.1% (77.8% to 73.7%) fully supervised and 3.3% (73.5% to 70.2%) weakly supervised; retaining all or only nonnegative connections also underperforms.The alternatives retaining all connections and removing only negative connections reduce accuracy by 1.9% and 2.3%, and by 1.8% and 2.3%, respectively, across the two settings.Source: §5.2 · p. 6
• Adding self-attention before the Bi-LSTMs decreases performance by 2.4% fully supervised and 2.7% weakly supervised.The authors speculate that PSP captures cross-modal relations while implicitly revealing intra-modal correlations.Source: §5.2 · p. 6
• The pair-similarity loss improves fully supervised accuracy by 1.2% on PSP and 1.5% on AVEL, while the weighting branch improves weakly supervised accuracy by 1.9% and 2.3%, respectively.Source: §5.2 · p. 6
• Accuracy remains stable as τ varies from 0 to 0.115, with the highest accuracy at τ = 0.095.The proportion of segments cut off depends on the video; videos containing more background may lose more content at the same threshold.Source: §5.2 · p. 7
• 77.8% fully supervised and 73.5% weakly supervised accuracy surpass AVEL by 9.2% and 6.8%, respectively.Source: Table 4 · p. 7, §5.2 · p. 7

5.3. Qualitative analysis

• For frying-food localization, the method attends to broader sound-related regions and associates the first four visual components with the first three audio components, unlike AVEL's missed segments.The visualized similarity matrices retain only a small proportion of connections after PSP selection.Source: §5.3 · pp. 7–8, Figure 4 · p. 7
• After PSP, audio and visual features cluster more cohesively within classes and diverge between classes than CNN or Bi-LSTM features.The t-SNE visualization uses validation-set features under the fully supervised setting.Source: §5.3 · p. 8, Figure 5 · p. 8

• PSP identifies relevant, including unsynchronized, audio-visual samples to enrich encoded features.Source: §6 · p. 8
• Negative and weak connections can harm the system, so the method removes them completely.Source: §6 · p. 8
• The fully supervised setting adds a pair-similarity loss, while the weakly supervised setting adds a weighting branch for temporal importance.Source: §6 · p. 8
• Extensive experiments validate the effectiveness of the proposed techniques.Source: §6 · p. 8