PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Positive Sample Propagation along the Audio-Visual Event Line

Jinxing Zhou, Dan Guo, Meng Wang

arXiv:2211.09980v1cs.CV

TL;DR

AVE localization must find event-containing video segments and identify their categories, despite audio-visual fusion methods being the prevailing focus. CPSP uses label-informed contrastive learning at pair, segment, and video levels; the authors report state-of-the-art performance under both full and weak supervision.

  • Problem

    AVE localization requires segment-level event and category predictions, while prior work has mainly focused on audio-visual feature fusion.

    Source: §1 · p. 1

  • Method

    CPSP uses full or weak labels to guide contrastive positive-sample propagation for audio-visual pairs and activation for segments and videos.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The authors report new state-of-the-art performance under both fully and weakly supervised AVE localization.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Pair-, segment-, and video-level positive-sample constraints are reported to benefit classifier training through refined audio-visual representations.

    Source: §7 · pp. 16–17

    In weak supervision, segment-level event labels are unknown and only video-level labels are available.

    Source: §3 · p. 4

Abstract

from arXiv · show

Visual and audio signals often coexist in natural environments, forming audio-visual events (AVEs). Given a video, we aim to localize video segments containing an AVE and identify its category. It is pivotal to learn the discriminative features for each video segment. Unlike existing work focusing on audio-visual feature fusion, in this paper, we propose a new contrastive positive sample propagation (CPSP) method for better deep feature representation learning. The contribution of CPSP is to introduce the available full or weak label as a prior that constructs the exact positive-negative samples for contrastive learning. Specifically, the CPSP involves comprehensive contrastive constraints: pair-level positive sample propagation (PSP), segment-level and video-level positive sample activation (PSA$_S$ and PSA$_V$). Three new contrastive objectives are proposed (\emph{i.e.}, $\mathcal{L}_{\text{avpsp}}$, $\mathcal{L}_\text{spsa}$, and $\mathcal{L}_\text{vpsa}$) and introduced into both the fully and weakly supervised AVE localization. To draw a complete picture of the contrastive learning in AVE localization, we also study the self-supervised positive sample propagation (SSPSP). As a result, CPSP is more helpful to obtain the refined audio-visual features that are distinguishable from the negatives, thus benefiting the classifier prediction. Extensive experiments on the AVE and the newly collected VGGSound-AVEL100k datasets verify the effectiveness and generalization ability of our method.

1 INTRODUCTION

AVE localization must match audio and visual signals, identify event-containing segments, and classify their categories in unconstrained videos. CPSP uses contrastive positive-sample propagation and activation at pair, segment, and video levels to refine representations.

1 INTRODUCTION

  • CPSP applies contrastive constraints to positive audio-visual pairs, event-containing segments, and same-category videos to encourage correspondence-, event-, and category-aware representations.The method covers both intra- and inter-video correlations.Source: §1 · pp. 2–3
  • The paper introduces a PSP-based pair-similarity loss for full supervision and a temporal weighting branch for weak supervision; both improvements also apply to other localization networks.The authors evaluate these improvements on the standard AVE dataset and report their effectiveness.Source: §1 · p. 3
  • PSP aggregates the most relevant audio-visual features, which need not be synchronized, while excluding weak or negative pair connections.The authors state that negative and weak connections can harm the system even when assigned small weights.Source: §1 · pp. 2–3
  • PSA adds segment-level contrast between AVE and background segments and video-level contrast between videos of different event categories.These constraints target event awareness and category awareness, respectively.Source: §1 · pp. 2–3
  • The authors report that CPSP achieves new state-of-the-art performance under both fully and weakly supervised AVE localization.They also report that the proposed backbone improvements benefit both settings and other localization networks.Source: §1 · p. 3
  • The paper also studies self-supervised PSP, releases VGGSound-AVEL100k with segment annotations, and evaluates CPSP on the related, more challenging LLP task.The authors report that LLP results demonstrate effectiveness and generalization ability.Source: §1 · p. 3

2 RELATED WORK

Prior AVE localization work centers on fusing audio and visual features, while related fields study correspondence, sound-source localization, and contrastive representation learning. CPSP is presented as a contrastive approach that selects relevant pairs, segments, and videos.

2 RELATED WORK

  • Audio-visual correspondence methods judge whether an image and short audio recording describe the same object, whereas AVE localization judges synchronized audio-visual segments over time.The paper draws on correspondence similarity to explore abundant audio-visual segment pairs in AVE localization.Source: §2 · p. 3
  • Sound-source localization identifies visual regions relevant to audio, while AVE localization locates events temporally; multiple-source localization must distinguish among possible sound-makers.Prior contrastive SSL methods construct positive and negative pairs, motivating the paper's use of available AVE instances.Source: §2 · pp. 3–4
  • AVEL predicts event presence and category for each segment, with existing methods using feature concatenation, bilinear fusion, co-attention, or self-attention.CPSP instead explores relevant pairs, event/background segments, and videos with similar but different categories.Source: §2 · p. 4
  • Audio-visual contrastive methods often treat synchronized segments as positives, but AVE localization can regard unsynchronized segments as positives when they depict the same event.The paper uses segment- and video-level label priors to construct contrastive samples and presents this as the first use of contrastive learning for AVE localization.Source: §2 · p. 4

3 PROBLEM STATEMENT

AVE localization predicts whether each synchronized audio-visual segment contains an event and, if so, its category. The paper considers fully supervised labels for every segment and weak supervision with video-level labels only.

3 PROBLEM STATEMENT

  • A video sequence is divided into T continuous, non-overlapping segments, each one second long, with visual and audio components.The task predicts event presence and category for each synchronized pair.Source: §3 · p. 4
  • Under full supervision, every segment has an event label indicating whether it is background or which event category it contains.The full video labels therefore identify the status of each synchronized audio-visual pair.Source: §3 · p. 4
  • Under weak supervision, the available label is video-level and represents the proportion of segments containing an event, while each segment's event label is unknown.This makes the setting more challenging than full supervision.Source: §3 · p. 4

4 OUR METHOD

The system encodes audio and visual sequences, propagates features through threshold-selected positive cross-modal connections, then refines them with segment- and video-level constraints before classification.

4 OUR METHOD

  • The overall pipeline orders feature encoding, PSP, PSA, and classification, with PSA applying segment- and video-level contrastive constraints to the fused features.The final classifier predicts whether segments contain an event and its category.Source: §4 · pp. 4–5, Figure 3 · p. 5

4.1 Feature extraction and encoding

  • AVGA focuses visual regions relevant to audio, and independent Bi-LSTMs encode temporal relations in the visual and audio sequences.The resulting encoded features are sent to PSP.Source: §4.1 · p. 5, §4 · p. 5

4.2 Positive sample propagation (PSP)

  • PSP connects audio-visual segment pairs by cross-modal similarity, prunes weak and negative connections with thresholding, then aggregates features through the remaining positive links.The resulting propagated features are further refined by PSA before classification.Source: §4.2 · pp. 5–6, §4 · p. 5

4.3 Positive sample activation (PSA)

PSA refines fused audio-visual features with contrastive constraints at segment and video levels: separating event segments from background and bringing same-category videos together.

4.3 Positive sample activation (PSA)

  • PSA first fuses the propagated audio and visual features into an integrated feature used to represent video segments and summarize the video.The resulting representations support segment- and video-level contrastive strategies.Source: §4.3 · p. 6

4.3.1 Segment-level positive sample activation (PSAS).

  • PSAS treats other event segments in a video as positives and background segments as negatives, pulling positives toward an event-segment anchor and pushing negatives away.The segment-level objective uses labeled event and background segments.Source: §4.3.1 · p. 6

4.3.2 Video-level positive sample activation (PSAV ).

  • PSAV selects the farthest same-category video as a positive and the top-K nearest different-category videos as hard negatives for each anchor.The objective draws the positive toward the anchor and pushes the selected negatives farther away.Source: §4.3.2 · p. 6

4.4 Classification

The classification module predicts segment categories in the fully supervised setting and weights segments to form a video-level prediction in the weakly supervised setting, with contrastive losses added where labels permit.

4.4.1 Classifier

  • In weak supervision, an added FC weighting branch assigns importance to temporal segments, and the weighted features produce the video-level prediction.The branch is intended to help distinguish event-containing segments when segment labels are unavailable.Source: §4.4.1 · p. 7

4.4.2 Objective function

  • Fully supervised training combines segment-level cross-entropy with an audio-visual pair similarity loss that encourages similar features for event-containing audio-visual pairs.The pair loss uses segment event labels, so it is unavailable in the weakly supervised setting.Source: §4.4.2 · pp. 7–8
  • The video-level positive-sample activation loss can be used in both supervision settings because both provide video-level event labels.The segment-level activation loss requires segment labels and is not applicable to weak supervision.Source: §4.4.2 · p. 8

5 DISCUSSION

The discussion explains how propagated features and contrastive objectives shape audio-visual representations, and why temporal weighting is used when segment supervision is absent.

5 DISCUSSION

  • PSP aggregates the most relevant positive visual features for each audio component, changing audio and visual feature representations through cross-modal support.The discussion describes the resulting adjustment in audio representation as verifiable in Fig. 9.Source: §5 · p. 8
  • Without segment labels, average pooling gives temporal segments similar weights; the added sigmoid weighting branch is intended to focus on segments containing an event.The branch is presented as helping the model distinguish temporal sequences and locate event-containing segments.Source: §5 · p. 8
  • The pair loss supplies event-presence supervision for synchronized audio-visual pairs, while segment- and video-level losses use intra-video and inter-video positive clues, respectively.Pair- and segment-level losses require segment labels; the video-level loss does not.Source: §5 · p. 8

6 EXPERIMENT

Experiments evaluate CPSP on localization benchmarks, through ablations and alternative training, and in video parsing and AVVP; visualizations examine feature propagation and segment- and video-level classifications.

6.5 Quantitative analysis - generalization to AVVP task

  • Experiments span AVE and VGGSound-AVEL100k localization and extend CPSP to LLP video parsing using HAN with weak video-level PSAV supervision.For multi-event videos, the extension uses a co-occurrence ratio to define positive samples based on shared event labels.Source: §6.1 · pp. 8–9, §6.5 · pp. 13–14

6.2 Comparison with state of the arts

  • On AVE, CPSP exceeds AVEL by 10.0% and 7.5% in fully and weakly supervised settings, respectively, and surpasses CMRA and PSP across both datasets and settings.The paper reports that CPSP remains robust on the larger VGGSound-AVEL100k benchmark, where all methods show a performance drop relative to AVE.Source: Table 1 · p. 9, §6.2 · pp. 9–10

6.3.1 Evaluation of the proposed PSP module

  • Removing PSP lowers AVE accuracy by 4.1% (77.8% to 73.7%) and 3.3% (73.5% to 70.2%) in fully and weakly supervised settings.Using all connections, retaining weak connections, or adding self-attention also reduces accuracy; the reported drops are 1.9%/2.3%, 1.8%/2.3%, and 2.4%/2.7%, respectively. Accuracy is generally stable for τ from 0 to 0.115 and peaks at τ = 0.095.Source: Table 2 · p. 10, §6.3.1 · p. 10

6.3.2 Evaluation of the proposed PSA module

  • PSA_S increased event-background centroid distances in 24 of 28 AVE categories, with increases of around 33% for female, guitar, and bus.This indicates improved separation of event and background segment features.Source: §6.3.2 · p. 10, Figure 6 · p. 11
  • With K = 4 and θ = 0.6, PSAV achieves 78.31% fully supervised and 74.20% weakly supervised accuracy; it improves predictions in 26 of 28 background-free categories.K = 4 is optimal among the tested values, and almost all tested PSAV setups exceed PSP’s 77.8% and 73.5% baseline accuracies.Source: §6.3.2 · pp. 10–11, Table 4 · p. 11, Figure 7 · p. 11

6.4.1 Comparison of the PSP and CPSP

  • On VGGSound-AVEL100k, PSAS and PSAV separately improve fully supervised PSP accuracy by 1.3% and 1.5%, while separate PSAS-then-PSAV training reaches 78.6% on AVE and 59.9% on VGGSound-AVEL100k.CPSP scores better than PSP on all reported clustering metrics and measurement methods, indicating improved feature clustering for event-background separation and event classification.Source: Table 5 · p. 12, §6.4.1 · pp. 12–13, Table 7 · p. 12

6.3.3 Evaluation of the improvement in fully/weakly setting

  • Adding Lavpsp improves fully supervised accuracy by 1.2% for PSP and 1.5% for AVEL, while the weighting branch improves weakly supervised accuracy by 1.9% and 2.3%, respectively.The weighting branch assigns different weights to temporal sequences, benefiting localization of target segments; both improvements also work on AVEL.Source: §6.3.3 · pp. 11–12, Table 6 · p. 12

6.4.2 Comparison of the CPSP and SSPSP

  • On VGGSound-AVEL100k, CPSP surpasses SSPSP by 1.1% and 0.4% in fully and weakly supervised settings, while their AVE performance is comparable.The paper attributes this difference to SSPSP ignoring semantic alignment when forming positive-negative pairs, which can produce false negatives for same-event pairs at different timestamps.Source: Table 9 · p. 13, §6.4.2 · p. 13

6.5 Quantitative analysis - generalization to AVVP task

  • On AVVP, CPSP surpasses AVEL and AVSDN, improves over PSP with Lvpsa, and outperforms HAN, especially on visual and audio-visual event F-scores.Against PSP, Type@AV and Event@AV improve by 3.9% and 3.3% at the segment level, and by 4.1% and 4.3% at the event level, respectively; the results provide further evidence of generalization to AVVP.Source: §6.5 · p. 15

6.6.1 Visualization of the effectiveness of PSP

  • PSP selects event-related audio-visual connections, improving attention to sound-relevant visual regions and producing more discriminative, better-clustered features.Compared with AVEL, the example shows attention to both frying chicken thighs and the pot, and PSP can use visual and audio components from different timestamps. Features become more cohesive within classes and divergent between classes after PSP.Source: §6.6.1 · pp. 15–16

6.6.2 Visualization of the effectiveness of CPSP

  • CPSP correctly classifies event-containing segments that PSP labels as background in both fully and weakly supervised examples.Its segment features outperform PSP on all reported discriminability indicators in both settings, attributed to PSAS aggregating event-aware semantics.Source: Figure 10 · p. 15, §6.6.2 · p. 16
  • In weakly supervised examples, PSAV helps CPSP correct a video-level category error and assign higher probabilities to correct segment labels.The examples show CPSP correcting a Helicopter prediction to Aircraft and assigning higher Car probabilities than PSP, distinguishing the category from similar Truck or background segments.Source: §6.6.2 · p. 16, Figure 11 · p. 16

7 CONCLUSION

CPSP learns distinguishable audio-visual representations by propagating positive samples at pair, segment, and video levels. Its contrastive constraints refine features for classifier training and are studied under fully, weakly, and self-supervised settings.

7 CONCLUSION

  • CPSP explores pair-, segment-, and video-level positive samples to learn distinguishable audio-visual representations.Pair-level PSP identifies the most relevant audio-visual pairs; segment-level PSA_S and video-level PSA_V add further contrastive constraints.Source: §7 · pp. 16–17
  • Negative and weak connections can harm the system even at small weights, so CPSP removes them completely.Source: §7 · p. 17
  • PSA_S separates event-containing segments from background, while PSA_V contrasts videos by event category; both benefit classifier training.The paper also collects VGGSound-AVEL100k and studies contrastive learning under fully, weakly, and self-supervised settings.Source: §7 · p. 17

APPENDIX A INTRODUCTION OF THE VGGSOUND-AVEL100K DATASET.

VGGSound-AVEL100k is a new dataset for evaluating audio-visual event localization models. It pairs video-tag category labels with manually annotated temporal audio-visual correspondence labels.

APPENDIX A INTRODUCTION OF THE VGGSOUND-AVEL100K DATASET

  • VGGSound-AVEL100k contains 101,072 videos across 141 event categories sampled from VGGSound.Source: Appendix A · p. 19
  • Category labels come from video tags, while annotators label each segment for simultaneous audio-visual events or their absence.A random 60% of annotated data is manually checked again for quality.Source: Appendix A · p. 19

B.1 Multi-run experiments of CPSP

Across five runs on AVE and VGGSound-AVEL100k, CPSP outperforms the compared methods and maintains relatively low accuracy variance. On VGGSound-AVEL100k, its average accuracy exceeds PSP in both supervision settings.

B.1 Multi-run experiments of CPSP

  • CPSP beats PSP and CMRA, while vanilla PSP outperforms most compared state-of-the-art methods across the reported cases.The reproduced baselines include AVEL, AVSDN, CMRA, and CMAN.Source: §B.1 · p. 19
  • CPSP maintains high performance with relatively low accuracy variance across the five runs on AVE and VGGSound-AVEL100k.The reported results use the best performance on each dataset for consistency with public literature.Source: §B.1 · p. 19
  • 1.9% (57.8% to 59.7%) fully supervised and 1.1% (47.1% to 48.2%) weakly supervised average-accuracy gains over PSP on VGGSound-AVEL100k.The comparisons are reported across five runs.Source: Table 10 · p. 19, §B.1 · p. 19

B.2 More discussion on the comparison to self-supervised methods

The comparison finds that self-supervised positive sample propagation can benefit from many negative samples but is sensitive to data distribution and batch size. CPSP is more stable across these conditions and achieves results comparable to Global-Local with fewer resources.

Comparison to the SSPSP

  • On AVE, SSPSP uses approximately 1537 times as many segment-level negative samples as CPSP, but can classify semantically corresponding pairs from different timestamps as negatives.Source: §B.2 · p. 20
  • SSPSP can improve AVE localization using numerous audio-visual pairs, but its performance is more sensitive to data distribution and training batch size.The authors summarize CPSP as superior and robust under these factors.Source: §B.2 · p. 20
  • SSPSP performs worse on pure-AVE Dae videos than on mixed AVE-and-background Dbg videos, while CPSP is nearly unaffected across the subsets.On VGGSound-AVEL100k, the cited SSPSP comparison drops from 58.8% to 50.2% with Dae.Source: Table 11 · p. 20, §B.2 · p. 20
  • With batch size 32 instead of 128, SSPSP accuracy on VGGSound-AVEL100k falls from 58.8% to 55.8%, while CPSP remains stable.Source: Table 12 · p. 20, §B.2 · p. 20

Comparison to the self-supervised method Global-Local

  • 82.3% vs. 82.1% fully supervised, CPSP with PSA_S and PSA_V versus Global-Local; 78.7% vs. 79.8% weakly supervised, CPSP with only PSA_V versus Global-Local.CPSP achieves these comparable results without extra data, using 1 FPS and one GTX 1080 GPU, compared with Global-Local's extra K-AV-240K pretraining and 16 Tesla P100 GPUs.Source: §B.2 · pp. 20–21, Table 13 · p. 21

APPENDIX C EXPERIMENTS FOR AUDIO-VISUAL VIDEO PARSING.

The AVVP positive-sample threshold performs best at µ = 0.6; the authors use this setting as a flexible choice for training. The ablation evaluates L_vpsa on LLP.

APPENDIX C EXPERIMENTS FOR AUDIO-VISUAL VIDEO PARSING.

  • CPSP achieves its best performance when the positive-sample threshold µ is 0.6.The authors use µ = 0.6 in their AVVP experiments.Source: Appendix C · p. 21
  • At µ = 1.0, positive videos must have exactly the same label set of multiple instance categories, a strict condition for batch training.The authors describe µ = 0.6 as 60% category coverage and a more flexible setting for selecting positives.Source: Appendix C · p. 21
  • The threshold ablation evaluates L_vpsa positive- and negative-sample construction for AVVP on the LLP dataset.Source: Table 14 · p. 21

Related papers

Loading 2211.09980v1…