PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning

Elad Amrani, Rami Ben-Ari, Daniel Rotman, Alex Bronstein

arXiv:2003.03186v3cs.CV

TL;DR

Expensive annotation and noisy associations limit self-supervised multimodal learning. This paper estimates noise through multimodal density and uses the estimates in a representation-learning building block, reporting comparable state-of-the-art performance across five datasets and two tasks.

  • Problem

    Multimodal annotation is challenging and expensive, while self-supervised multimodal systems face high noise that can yield sub-optimal results.

    Source: §1 · p. 1

  • Method

    The method reduces multimodal noise estimation to density estimation and uses inherent cross-modal correlation in a noise-robust representation-learning building block.

    Source: §1 · p. 1

  • Results

    Comparable state-of-the-art performance is reported on five datasets for Video Question Answering and Text-to-Video Retrieval.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The building block is presented as integrable into many multimodal learning models, with Soft Max Margin as one demonstrated integration.

    Source: §7 · p. 7

    In a manual inspection of 400 HowTo100M clip-caption pairs, about half had no caption-mentioned object or action visually present.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

One of the key factors of enabling machine learning models to comprehend and solve real-world tasks is to leverage multimodal data. Unfortunately, annotation of multimodal data is challenging and expensive. Recently, self-supervised multimodal methods that combine vision and language were proposed to learn multimodal representations without annotation. However, these methods often choose to ignore the presence of high levels of noise and thus yield sub-optimal results. In this work, we show that the problem of noise estimation for multimodal data can be reduced to a multimodal density estimation task. Using multimodal density estimation, we propose a noise estimation building block for multimodal representation learning that is based strictly on the inherent correlation between different modalities. We demonstrate how our noise estimation can be broadly integrated and achieves comparable results to state-of-the-art performance on five different benchmark datasets for two challenging multimodal tasks: Video Question Answering and Text-To-Video Retrieval. Furthermore, we provide a theoretical probabilistic error bound substantiating our empirical results and analyze failure cases. Code: https://github.com/elad-amrani/ssml.

1 Introduction

Multimodal self-supervision can use abundant unlabeled data, but high noise in associated video-text pairs limits performance. The paper estimates noise through multimodal density and evaluates the resulting building block across five datasets and two tasks.

1 Introduction

  • Manual annotation of multimodal datasets is challenging and expensive, limiting supervised learning progress and making annotation a key development bottleneck.Self-supervised learning seeks to use abundant unlabeled web data, while many systems still use only one modality.Source: §1 · p. 1
  • Self-supervised multimodal systems use correlations in unlabeled instructional videos, but high noise levels can yield sub-optimal results.The paper proposes a noise-robust representation-learning building block based on the inherent correlation between modalities, reducing noise estimation to multimodal density estimation.Source: §1 · p. 1
  • Comparable state-of-the-art performance is reported on five datasets across Video Question Answering and Text-to-Video Retrieval.The evaluated datasets are MSRVTT, LSMDC, MSVD, MSRVTT-QA, and MSVD-QA.Source: §1 · pp. 1–2
  • The paper also provides a theoretical probabilistic error bound and analyzes failure cases to substantiate its empirical results.Its contributions include integrating the building block into the max margin ranking loss function.Source: §1 · pp. 1–2

2 Related Work

Prior work motivates self-supervised multimodal learning and describes approaches to representation learning, density estimation, and noisy labels. This paper focuses on coordinated representations and multimodal k-nearest-neighbor density estimation.

2 Related Work

  • Speech-visual correspondence in unlabeled instructional videos is a noisy pretext task that can provide visual features with semantic information in natural text.The paper focuses on this task, with speech converted to text using automatic speech recognition.Source: §2 · p. 2
  • Multimodal representation frameworks include joint, encoder-decoder, and coordinated representations; this work focuses on coordinated representations that enforce similarity.The goal is to bring representations of similar concepts close together.Source: §2 · p. 2
  • The paper uses multimodal k-nearest-neighbor density estimation, a special case of kernel density estimation, to form a noise-robust representation-learning model.Density estimation aims to estimate the probability density underlying data assumed to be independently and identically distributed.Source: §2 · p. 2
  • Existing noisy-data methods model noise or correct labels, but the paper says they do not address multimodal association labels and often require clean labels.Its proposed true-label estimation for multimodal data instead relies on correlation between modalities without requiring clean labels.Source: §2 · p. 2

3 Method

The method treats semantic mismatches between clips and captions as noise and estimates pair reliability through multimodal density. It then uses that estimate to weight a max margin ranking loss.

3.1 Motivation

  • About half of 400 randomly sampled HowTo100M clip-caption pairs had no caption-mentioned object or action visually present, motivating noise estimation from modality correlation.The method assumes videos and captions can each be represented by a mixture model of concepts.Source: §3.1 · pp. 2–3

3.2 Notation and Problem Formulation

  • The formulation maps video and caption embeddings into a common space, where correctly associated pairs should have similar representations.Each clip-caption pair has an indicator for whether the association is correct or wrong.Source: §3.2 · p. 3

3.3 Noise Estimation Using Multimodal Density Estimation

  • A pair is close only when both its video and caption are close to the corresponding modalities in another pair; local k-NN density yields a normalized estimated probability of correct association.The similarity function normalizes each modality's similarity values before taking their minimum.Source: §3.3 · p. 3

3.4 Soft Max Margin Ranking Loss

  • Soft Max Margin Ranking weights each associated pair by its estimated probability of being correct while ranking video-caption matches against negative pairs.Discarding samples below a probability threshold was another integration option, but performed substantially worse in real-data experiments.Source: §3.4 · p. 3

4 Theoretical Analysis

The paper bounds the probabilistic error of multimodal noise estimation under a Gaussian-mixture model, then uses simulations to examine parameter effects and failure cases. The simulations identify a transition in K and a small-concept failure mode.

4.1 Probabilistic Error Upper Bound

  • The probabilistic error upper bound assumes Gaussian-mixture data and considers scalar pairs across T>1 equally probable concepts.The analysis assumes scalar pairs for simplicity, but imposes no assumption on concept placement beyond 6σ-separation; a toy dataset corroborates validity in multiple dimensions.Source: §4 · p. 3, §4.1 · pp. 3–4
  • For each pair, P=1 denotes correct association, with probability 1−η, where η is the dataset noise ratio.The analysis defines the average similarity of each sample to its K nearest neighbors using the sum of within-modality similarities.Source: §4.1 · p. 4

4.2 Numerical Simulations and Analysis

  • The simulations set τ* to equalize the two conditional error probabilities, sweep one parameter at a time, and optimize t over [1, 100].The swept parameters are K, η, M, and T.Source: §4.2 · p. 4, Figure 2 · p. 5
  • Error-bound simulations identify K0≈M/T·(1−η): performance is good when K>K0 and begins to fail when K≤K0.The bound goes to zero as dataset size M increases; the authors note that a global K is suboptimal when real-world concepts are not equi-probable.Source: §4.2 · p. 4, Figure 2 · p. 5
  • The method can fail for small T regardless of K0, because wrongly associated pairs may coincide across modalities; this problem is alleviated by a factor of O(T^2) as T increases.The authors note that T is almost always large in real-world data.Source: §4.2 · pp. 4–5

5 Experimental Settings

The experiments train on HowTo100M and use shared multimodal embeddings for downstream video question answering and text-to-video retrieval. The section describes feature construction and evaluation datasets, plus a toy visualization of estimated association probabilities.

5.1 Implementation Details

  • The model trains on 1.16M available HowTo100M videos, pairing timestamped captions with clips extended to at least five seconds.The full dataset contains more than 1.2M videos; some were unavailable for download.Source: §5.1 · p. 5
  • Visual features combine ImageNet-pretrained ResNet-152 frames at 1 fps with Kinetics-pretrained ResNeXt-101 16-frame features at 24 fps, followed by temporal max pooling.Word representations use pretrained GoogleNews word2vec; sentence representations average the word vectors.Source: §5.1 · p. 5

5.2 Downstream Tasks

  • Video VQA is evaluated on MSRVTT-QA and MSVD-QA using learned question-plus-video and answer embeddings, max-margin ranking, and nearest-neighbor answer retrieval.The learned embeddings are fine-tuned from HowTo100M representations.Source: §5.2 · p. 5
  • Text-to-video retrieval uses nearest-neighbor search in the joint embedding space and is evaluated on MSRVTT, MSVD, and LSMDC with R@1, R@5, R@10, and median recall.Source: §5.2 · pp. 5–6
  • In the toy dataset, the input embeddings are not easily separated by association status, while threshold 0.48 yields approximately 0.9 precision and recall.The figure shows t-SNE embeddings and estimated-probability cumulative distributions.Source: Figure 3 · p. 6

6 Experiments and Analysis

The toy experiment illustrates the method and corroborates the theoretical analysis in the multidimensional case. Benchmark evaluation reports gains over the self-supervised baselines across five datasets and two tasks, with additional state-of-the-art and 2D-feature comparisons.

6 Experiments and Analysis

  • The experiment section compares the proposed model with two ablative baselines and task-specific state-of-the-art models.The section overview also identifies the toy dataset, design-choice analysis, and qualitative examples as additional analyses.Source: §6 · p. 6, §6.2 · p. 6

6.1 Multidimensional Toy Dataset

  • With η=0.5, M=1250, T=50, and k=4, the toy experiment visualizes embeddings and estimated correct-association probabilities, corroborating Theorem 1 in the multidimensional case.The embeddings are visualized with t-SNE, and the estimated probabilities with an empirical cumulative distribution function.Source: §6.1 · p. 6

6.2 Ablative Baselines and SOTA Models

  • HTM-PT uses the same architecture and training as the proposed model but replaces Soft Max Margin with max margin ranking loss; HTM-no-PT omits HowTo100M pretraining.The authors use these as ablative baselines.Source: §6.2 · p. 6
  • Our model consistently outperforms HTM-PT and HTM-no-PT on Video Question Answering and Text-To-Video Retrieval across five datasets, setting new state of the art on MSRVTT-QA and MSVD-QA.The results are reported in the Text-To-Video Retrieval and Video Question Answering tables.Source: §6.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
  • Text-To-Video Retrieval reaches new state-of-the-art performance on MSRVTT and MSVD; without MIL-NCE's additional clip-caption pairs, our model outperforms it on MSRVTT.The authors hypothesize MIL-NCE's advantage is related to its end-to-end training and use of adjacent clip-caption pairs.Source: §6.2 · p. 7
  • Without 3D features, the model remains on par with HTM-PT, allowing training and inference with only 2D features without or with minor performance degradation.The comparison is described as a disadvantaged setting relative to training with 2D+3D features.Source: §6.2 · p. 7

7 Summary

The paper frames multimodal noise estimation as density estimation and presents a noise-robust representation-learning building block. It reports comparable state-of-the-art results across five datasets and two tasks, supported by probabilistic analysis.

7 Summary

  • Noise estimation for multimodal data is reduced to multimodal density estimation, yielding a building block for noise-robust representation learning.The block is integrated into Soft Max Margin and is described as applicable to other multimodal models and losses.Source: §7 · p. 7
  • The method is trained on noisy speech-visual correspondence and evaluated on Video Question Answering and Text-to-Video Retrieval across five datasets.The authors provide a probabilistic error bound to support the empirical results and analyze failure cases.Source: §7 · p. 7

Appendix Overview

  • The appendices contain the proof and toy-data corroboration of Theorem 1, qualitative examples, a low-T failure-case visualization, a 2D-feature comparison, design-choice evaluations, and implementation details.Source: Appendix Overview · p. 10

A Proof of Theorem 1

The proof derives a moment-generating function for the average similarity score conditional on association status, then applies a Chernoff bound. Numerical simulations are used to make the resulting error bound interpretable.

A Proof of Theorem 1

  • The proof starts from the moment-generating functions of the two modality-specific similarities, noting that their negatives follow folded normal distributions.It then derives the moment-generating function for the average similarity score.Source: A Proof of Theorem 1 · p. 10
  • Conditioning on the number of pairs from each component, the proof uses total expectation and multiplies moment-generating functions for independent weighted variables.The component-pair count is denoted m̃_i, and the derivation invokes the three-sigma limit assumption.Source: A Proof of Theorem 1 · p. 10
  • The proof concludes with a Chernoff bound; simulations set τ=τ* to equalize the two conditional error probabilities and repeat each experiment 10 times.The simulation setup also optimizes t over [1, 2, ..., 100].Source: A Proof of Theorem 1 · p. 10

B Reproducing Empirically the Theoretical Results

Multidimensional simulations reproduce the theoretical trends, while qualitative examples illustrate score behavior and a low-concept-count failure case.

B Reproducing Empirically the Theoretical Results

  • On the multidimensional toy dataset, Figure 4 reproduces theoretical trends, corroborates Theorem 1 beyond one dimension, shows K0’s influence across panels 4a–4d, and exhibits the low-T failure case in 4e.The vertical axis is 1 − precision at the F1-maximizing threshold; the red dashed vertical line marks K = M/T·(1−η).Source: B Reproducing Empirically the… · p. 11, Figure 4 · p. 11

C.1 High Score Examples and Their k Nearest Neighbours

  • High-score 'bye' and 'mix' examples have nearest neighbors depicting waiving goodbye and mixing, respectively; these examples use noise scores, not the learned shared embedding.The displayed frames represent clips that contain the described actions.Source: §C.1 · p. 12

C.2 High and Low Score Examples for the Same Query

  • Ten same-query examples visually show high- and low-p̂ scores distinguishing correctly and wrongly associated clip-caption pairs.Source: §C.2 · p. 13

D Low T Failure Case Simulation

  • With small T, collisions among wrongly associated pairs from the same concept pair can cause failure regardless of K0; increasing T alleviates this problem by O(T^2).Source: D Low T Failure Case Simulation · p. 14

E Zero-shot Text-To-Video retrieval in ‘unfair’ settings

In zero-shot Text-To-Video retrieval under the stated ‘unfair’ settings, the 2D-only model is at least on par with HTM-PT. Using only 2D features can save time or computation, though the MSRVTT comparison warrants further investigation.

F Design-Choice Analysis

The design analysis finds a non-monotonic effect of K and no conclusive advantage for minimum over mean similarity. Implementation details specify negative sampling, neighbor search, and retrieval test splits.

F Design-Choice Analysis

G Additional Implementation Details

Loading 2003.03186v3…