PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Audio-Visual Instance Discrimination

Pedro Morgado, Ishan Misra, Nuno Vasconcelos

arXiv:2103.15916v1cs.CV

TL;DR

Cross-modal instance discrimination assumes paired audio and video are informative of each other and other instances are unrelated, but faulty positives and negatives violate those assumptions. The paper combines weighting for weak correspondences with soft targets for similar instances. Experiments report improved representation learning and transfer performance, including comparisons with prior work.

  • Problem

    Cross-modal instance discrimination assumes paired audio and video are informative of each other and other instances are unrelated, assumptions that fail for faulty positives and semantically similar negatives.

    Source: §1 · p. 1

  • Method

    The method down-weights weak audio-visual correspondences and optimizes an instance-discrimination loss with soft targets encoding relationships between instances.

    Source: §1 · p. 2, §4.2 · pp. 4–5

  • Results

    The proposed procedure enhances representation learning and improves transfer performance significantly.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The findings frame cross-modal learning as learning with noisy targets and identify weighting and soft targets as approaches to faulty positives and negatives.

    Source: §7 · p. 8

    The experiments focus on audio-visual learning, while other modality pairs are identified as a direction for future work.

    Source: §7 · p. 8

Abstract

from arXiv · show

We present a self-supervised learning method to learn audio and video representations. Prior work uses the natural correspondence between audio and video to define a standard cross-modal instance discrimination task, where a model is trained to match representations from the two modalities. However, the standard approach introduces two sources of training noise. First, audio-visual correspondences often produce faulty positives since the audio and video signals can be uninformative of each other. To limit the detrimental impact of faulty positives, we optimize a weighted contrastive learning loss, which down-weighs their contribution to the overall loss. Second, since self-supervised contrastive learning relies on random sampling of negative instances, instances that are semantically similar to the base instance can be used as faulty negatives. To alleviate the impact of faulty negatives, we propose to optimize an instance discrimination loss with a soft target distribution that estimates relationships between instances. We validate our contributions through extensive experiments on action recognition tasks and show that they address the problems of audio-visual instance discrimination and improve transfer learning performance.

1. Introduction

Cross-modal instance discrimination uses audio and video as paired views, but real-world correspondences and randomly sampled negatives can both be unreliable. The paper proposes weighting weak positives and using soft targets for faulty negatives, then evaluates the approach on downstream tasks.

1. Introduction

  • Audio-visual learning can contain faulty positives when modalities are uninformative of each other and faulty negatives when sampled instances are semantically related.Both violate assumptions that paired modalities are informative and that other instances are unrelated.Source: §1 · p. 1
  • The method weights contrastive loss to reduce weak-correspondence positives and uses a soft target distribution to account for similarity between instances.Instances with evidence of similarity may cease to act as negatives and may become positives.Source: §1 · p. 2, Figure 2 · p. 2
  • The authors report that an oracle removing faulty positives and negatives improves a state-of-the-art xID method, and that their proposed method is effective on several downstream tasks.The paper frames the two noise sources and the robust loss as its main contributions.Source: §1 · p. 2

2. Related work

Prior self-supervised work learns from data-defined pretext tasks, and cross-modal instance discrimination uses audio-visual correspondence as alternative views. This paper addresses faulty positives and negatives with soft-target instance discrimination rather than computationally costly clustering.

2. Related work

  • Self-supervised learning uses pretext tasks defined from data rather than human annotations, while cross-modal instance discrimination generates views from different modalities.The latter avoids the need for heavy augmentations to create distinct views in single-modality contrastive learning.Source: §2 · p. 2
  • Audio-visual methods use correspondence for tasks including same-instance classification, temporal synchronization, spatial alignment, and cross-modal instance discrimination [61].Cross-modal instance discrimination contrasts instances against many negatives and has shown downstream action-recognition performance [61].Source: §2 · p. 2
  • Unlike clustering approaches, the proposed soft-target instance discrimination addresses faulty negatives without significant clustering overhead.The paper connects its treatment of faulty correspondences to supervised learning with noisy labels, while distinguishing faulty positives and negatives as its noise sources.Source: §2 · pp. 2–3

3. Analysis: Instance Discrimination

Cross-modal instance discrimination aligns audio and video features from the same instance while separating other instances, but faulty correspondences distort its training signals. The analysis explains how weak positives and semantically similar negatives can exert disproportionately strong gradient forces.

3. Analysis: Instance Discrimination

  • Cross-modal instance discrimination uses contrastive loss to align audio and video features from the same instance, with target representations that may come from memory banks, network outputs, momentum encoders, or clustering centroids.This work builds on AVID with memory-bank targets.Source: §3 · p. 3
  • Faulty positives occur when audio and visual modalities are uninformative of each other, while faulty negatives are semantically similar samples that should ideally be positives.The paper notes these correspondences occur in practice for a significant number of instances.Source: §3 · p. 3
  • Lower audio-video similarity scores tend to indicate uninformative correspondences and faulty positives, while high video-to-negative similarity can indicate faulty negatives.Figure 3 presents a histogram of audio-video similarities and examples at different points in its distribution.Source: §3 · p. 3, Figure 3 · p. 4
  • Faulty positives receive stronger attraction gradients because their lower similarity yields less confident predictions, while faulty negatives exert strong repulsion through high matching posteriors.These gradient contributions can send noisy training signals even though similar negatives should be close in feature space.Source: §3 · pp. 3–4

4. Robust audio-visual representation learning

The section addresses faulty positives and negatives in cross-modal instance discrimination with two complementary mechanisms: weighting weak audio-visual correspondences and softening targets for similar instances. After an initial cross-modal instance-discrimination warmup, the mechanisms can be combined.

4. Robust audio-visual representation learning

  • The section motivates these mechanisms as responses to the overemphasis that standard contrastive learning places on faulty positives and negatives.Faulty positives arise from weak audio-visual correspondence; faulty negatives are addressed through instance-similarity soft targets.Source: §4 · p. 4, §4.2 · p. 4

4.1. Weighted xID: Tackling Faulty Positives

  • Low audio-visual similarity indicates potentially faulty positives, so weighted xID scales their loss contribution using weights proportional to the similarity-score cumulative distribution.The weights use a transformed normal distribution and soft truncation to assign low-score instances a non-zero minimum weight.Source: §4.1 · p. 4

4.2. Soft Targets: Tackling Faulty Negatives

  • Soft-xID replaces equally negative targets with similarity-based soft targets, reducing the repelling force on faulty negatives.The target mixes one-hot labels with softening scores; when λ = 0, it matches vanilla xID.Source: §4.2 · pp. 4–5
  • Softening-score strategies include bootstrapping, swapped prediction, neighbor prediction, and cycle-consistent prediction.Bootstrapping uses each model’s own posteriors; swapped prediction uses the opposite modality’s posteriors, while neighbor prediction uses within-modal relationships.Source: §4.2 · p. 5

4.3. Training

  • The training procedure begins with cross-modal instance discrimination alone, then combines sample weighting with the soft-target loss after warmup.The combined objective uses the sample weights w_i and LSoft-xID.Source: §4.3 · p. 5

5. Experiments

Experiments test whether weighting faulty positives and using soft targets for faulty negatives improve cross-modal representations. The combined Robust-xID method improves on each component, while the individual approaches address their respective sources of noisy training signals.

5. Experiments

  • Models were pretrained on a 50K-video subset of Kinetics-400 and evaluated by transfer learning; video representations were also evaluated through retrieval experiments without fine-tuning.For retrieval, 4×4×512 feature maps from 10 clips per video were averaged. Test-set videos queried the training set, and a retrieval counted as correct when a top-k result matched the query’s class.Source: §5 · p. 5, §5.1 · p. 6

5.2. Weighted cross-modal learning

  • As injected faulty-positive fractions increased, xID transfer performance decreased significantly, while Weighted-xID reduced their impact.The study replaced an instance’s audio with randomly selected audio not part of the training set; the unaltered pretraining data also contains an unknown but significant number of faulty positives.Source: §5.2 · p. 6, Figure 7 · p. 6
  • Weighted-xID outperformed xID in nearly all tested cases and even surpassed Oracle-xID, especially at lower injected-noise levels.The original Kinetics data already contains weak correspondences that Oracle-xID treats as clean, while the learned weighting can suppress them.Source: §5.2 · pp. 6–7

5.3. Instance discrimination with soft targets

  • Estimated soft targets significantly improved over xID, with cycle-consistent prediction most often the strongest strategy.It uses the opposite modality and avoids targets that are themselves poor cross-modal correspondences.Source: §5.3 · p. 7

5.4. Robust instance discrimination with soft targets

  • Robust-xID outperformed both Weighted-xID and Soft-xID, whose gains over vanilla xID were complementary.The combined loss addresses faulty positives and faulty negatives with separate mechanisms.Source: §5.4 · p. 7

6. Comparison to prior work

The paper compares Robust-xID with prior self-supervised methods on fine-tuning, retrieval, and few-shot tasks. It reports better performance than prior work under the stated compute conditions, including gains over CMA and methods using more compute.

6. Comparison to prior work

  • For the fine-tuning comparison, models were pretrained on Kinetics and evaluated on UCF and HMDB using top-1 video-level accuracy from 10 clips.The comparison table also lists video encoder, pretraining compute, and fine-tuning resolution.Source: §6 · p. 7, Table 3 · p. 8
  • Robust-xID outperformed prior work whose pretraining used a single node and methods such as SeLaVi that require 8× more compute.It also outperformed xID+CMA, which can partially address faulty negatives; the authors describe Robust-xID as a simpler online approach to both faulty positives and negatives.Source: Table 3 · p. 8, §6 · p. 8
  • Robust-xID significantly improved over prior work on both retrieval and few-shot learning on UCF and HMDB.Retrieval reports R@K for K = 1, 5, and 20; few-shot reports top-1 accuracy for n ∈ {1, 5, 20} examples per class.Source: §6 · p. 8, Table 4 · p. 8, Table 5 · p. 8

7. Discussion and future work

The paper frames audio-visual instance discrimination as learning with noisy targets, identifying faulty positives and faulty negatives as key sources of noise. It reports an annotation-free approach that improves transfer performance and points to noisy-correspondence methods and other modality pairs as future directions.

7. Discussion and future work

  • Weak audio-visual correspondences and semantically similar negatives are two significant sources of noisy training signals in audio-visual instance discrimination.The paper characterizes cross-modal learning as learning with noisy targets.Source: §7 · p. 8
  • The proposed method mitigates faulty correspondences without ground-truth annotations and significantly improves transfer performance.The paper reports extensive analysis and experimental evaluations supporting these outcomes.Source: §7 · p. 8
  • Future work could adapt noisy-correspondence methods beyond weighting and soft targets and investigate modality pairs such as RGB and flow or instructional-video text.The authors focused on audio-visual learning and describe these other modality pairs as potential extensions.Source: §7 · p. 8

A. Parametric studies

The parametric studies examine how the weighting shape parameter δ and Soft-xID mixing coefficient λ affect transfer performance. Performance is robust to δ when clear faulty positives are suppressed, while λ requires a moderate range: too much softening causes pretraining to fail.

Weight function shape parameter δ

  • Weighted-xID is not very sensitive to δ as long as clear faulty positives are suppressed.Figure 8 evaluates transfer performance on UCF and HMDB under full finetuning and retrieval protocols.Source: A. Parametric studies · p. 13, Figure 8 · p. 13

Soft-xID: Mixing coefficient

  • In Soft-xID, λ mixes one-hot instance-discrimination targets with softening scores, with λ = 0 retaining the one-hot targets.The study used cycle-consistent targets as the softening strategy.Source: A. Parametric studies · p. 13
  • λ between 0.25 and 0.5 yields optimal performance, while λ > 0.65 makes pretraining fail completely.At large λ, targets derive from model predictions alone and disregard instance labels.Source: A. Parametric studies · p. 13

B. Additional analysis

Additional analyses show that Robust-xID retrieves semantically related videos, with class-level retrieval varying alongside the distinctiveness and duration of accompanying sounds. A manual check of likely faulty positives found clear mismatches as well as voice-over cases and misidentified samples.

Retrieval

  • Robust-xID retrieves semantically related videos; errors can be intuitive, and retrieval failures appear correlated with classes that are hard to distinguish from audio alone.Figure 11 shows the query video and its top four retrievals in each row, using Euclidean distance between video feature maps.Source: B. Additional analysis · p. 13, Figure 11 · p. 15

Class-based analysis

  • Using 4 × 4 × 512 feature maps from a Robust-xID video encoder learned on full Kinetics, the authors report per-class R@1 averaged across images and sort classes by that score.Figure 10 shows the top 40 and bottom 40 classes.Source: B. Additional analysis · p. 13, Figure 10 · p. 14

Faulty positive detection performance

  • Of 100 videos sampled from the 10,000 most likely faulty positives, 67 were clear faulty pairs, 24 contained narrative voice-overs, and 9 were misidentified.The voice-over cases required natural-language understanding to link audio and video.Source: B. Additional analysis · p. 14

Related papers

Loading 2103.15916v1…