PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What to Trust in Multimodal Learning under Noisy Supervision

Jiashuo Zou, Xiaobo Xia

arXiv:2610.11057v1cs.CVcs.LG

TL;DR

Real-world multimodal classification faces potentially corrupted labels, while traditional sample selection has largely focused on unimodal data and may miss useful cues across representation spaces. REFINE analyzes representation structure and selects examples using fused and unimodal spaces. Experiments report consistent improvements in sample-selection quality and classification performance across diverse datasets, noise settings, and training paradigms.

  • Problem

    Multimodal classification methods typically rely on correct labels, while existing sample-selection methods primarily target unimodal settings despite fused and unimodal spaces offering distinct label-reliability cues.

    Source: §1 · pp. 1–2

  • Method

    REFINE constructs target–background discriminative eigenvectors, selects trusted fused or unimodal spaces by class, and retains examples through label agreement.

    Source: §1 · p. 2

  • Results

    Experiments across diverse datasets, noise settings, and training paradigms demonstrate consistent improvements in sample-selection quality and classification performance.

    Source: §6 · p. 16

  • Takeaways & Limitations

    REFINE provides more reliable supervision for multimodal classifier training by combining examples selected from trusted representation spaces.

    Source: §1 · p. 2

    The analysis frames class separation geometrically through the angle between the two class-mean directions.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Multimodal classification processes and relates information from multiple modalities to achieve more accurate predictions. However, existing methods typically rely on high-quality ground-truth labels, which are difficult to obtain in real-world scenarios. While sample-selection methods for learning with noisy labels aim to identify correctly labeled examples from noisy data, traditional methods primarily focus on unimodal settings and fail to exploit multimodal information fully. This motivates us to build a more reliable noise detector in multimodal learning. To this end, we theoretically analyze the relationship between representation structure and noise detection capability. Based on this analysis, we propose REFINE, which is a multimodal label-noise detection framework that jointly uses fused and unimodal representations for label-noise detection. Specifically, REFINE constructs discriminative eigenvectors through discriminative analysis of the target and background classes and selects trusted representation spaces with better noise detection capability for each class. Within each trusted space, REFINE measures the alignment between each instance representation and the discriminative eigenvectors. It then combines the subsets selected from these spaces. The combined set provides cleaner supervision for updating the multimodal classifier, thereby reducing the influence of mislabeled examples during training and improving model generalization. Extensive experiments across diverse tasks demonstrate REFINE's superiority compared to baseline methods. The source code will be publicly available.

1 Introduction

Multimodal noisy-label learning must identify which information can assess label reliability, because real-world labels may be corrupted and noisy-label memorization can hinder generalization. REFINE selects reliable examples using fused and unimodal representations, with experiments reporting improved performance across diverse settings.

Motivation

  • Real-world manual or web-crawled annotations may be corrupted, and even a small amount of corrupted data can hinder neural-network generalization through memorization of noisy labels.This motivates training more reliable multimodal classifiers under noisy supervision.Source: §1 · p. 1
  • Existing sample-selection methods primarily target unimodal settings, while fused-only selection may overlook modality-specific cues relevant to label reliability.The introduction frames multimodal noise detection as identifying useful information and using it to distinguish correctly labeled from mislabeled examples.Source: §1 · p. 2, Figure 1 · p. 2

Contributions

  • The paper theoretically relates representation structure to noise-detection capability, motivating class-wise selection of representation spaces with stronger detection ability.Its stated goal is reliable supervision for multimodal classification under noisy supervision.Source: §1 · p. 2
  • REFINE combines fused and unimodal representation spaces in one sample-selection process for multimodal noise detection.It analyzes target and background classes, selects trusted spaces by class, and uses label agreement to retain examples without predefined thresholds, noise-rate estimates, or extra clean examples.Source: §1 · pp. 2–3
  • Across seven datasets and multiple noise types, REFINE demonstrates superior classification performance, with analyses supporting its selection quality and design rationale.Experiments also report improvements when REFINE is integrated with different noisy-label training paradigms as a plug-in sample-selection mechanism.Source: §1 · p. 3

2 Related Work

Prior noisy-label methods include several training strategies, but sample-selection work has mainly focused on unimodal settings. Multimodal methods use varied assumptions and supervision, motivating selection that exploits fused and unimodal spaces without extra clean data or prior noise-rate knowledge.

2.1 Learning with Noisy Labels

  • Sample selection identifies correctly labeled examples to limit incorrect supervision, but existing approaches primarily focus on unimodal settings.FINE [12] uses latent structure within each observed class in a single representation space; multimodal models provide fused and unimodal spaces whose clean-noisy separation can vary by class.Source: §2.1 · p. 3
  • Noisy-label learning also includes noise-tolerance or regularization, transition modeling, label correction, and semi-supervised approaches.Sample selection is the focus here because it connects label reliability with learned representation structure.Source: §2.1 · p. 3

2.2 Multimodal Representation Learning and Classification

  • Multimodal classification commonly fuses modality-specific encodings into a shared representation, and this work focuses on feature-level fusion.Other paradigms include decision-level fusion.Source: §2.2 · p. 3
  • Prior multimodal noisy-supervision methods use cross-view consistency, extra clean data, transition modeling, or small-loss selection, while some assume individually predictive modalities or rely on noise-affected predictions.REFINE instead uses fused and unimodal spaces without additional clean samples or prior noise-rate knowledge.Source: §2.2 · p. 3

3 Preliminaries

The preliminaries formulate multimodal classification with potentially corrupted observed labels, recap FINE's alignment-based selection, and analyze how representation geometry affects its clean-noisy separation. The analysis motivates adapting selection to class-specific representation spaces.

3.1 Problem Formulation

  • The task is to select examples with reliable observed labels from a multimodal dataset and use them to train a more robust classifier.The formulation uses feature-level fusion, modality-specific representations, and a fused representation; its theoretical analysis considers binary labels and assumes Gaussian class-conditional representations with unit mean directions.Source: §3.1 · p. 4

3.2 FINE for Learning with Noisy Labels: A Recap

  • FINE [12] scores examples by alignment with the leading eigenvector of their observed class, then fits a two-component GMM to select the component with the larger mean as clean.It selects within a single representation space, leaving the choice of space open for multimodal models.Source: §3.2 · p. 4

3.3 Limitations of FINE in Multimodal Formulation

  • The alignment-score difference measures clean–mislabeled separation along a direction, and larger differences improve lower bounds on recall and precision.FINE's difference increases with the class angle and decreases with the mislabeled-to-clean ratio; because class angles vary by space, no representation type is optimal for every class.Source: §3.3 · p. 5

4 Method

REFINE uses target–background discriminative analysis to build class-specific eigenvectors, selects trusted fused or unimodal spaces per class, and unions examples selected by label agreement. Its theory explains how background information sharpens clean/noisy separation and how space comparisons identify trusted representations.

4 Method

  • REFINE constructs class-specific discriminative eigenvectors, selects trusted spaces from fused and unimodal representations, then unions reliable examples selected within those spaces.The framework comprises discriminative analysis, trusted space selection, and label agreement.Source: §4 · p. 5, Figure 2 · p. 6

4.1 Discriminative Analysis

  • REFINE weights non-target classes by alignment with the target’s leading direction, then finds an eigenvector maximizing target alignment relative to background, with ε regularization.Non-target classes with stronger alignment receive larger background weights; ε is added to the background matrix diagonal to address possible rank deficiency.Source: §4.1 · p. 6

4.2 Theoretical Analysis of DE

  • Incorporating background information increases the alignment-score difference between clean and mislabeled examples within the same space, and this advantage grows as contamination in the target observed class increases.Source: §4.2 · p. 7

4.3 Trusted Space Selection

  • For each class, REFINE compares spaces pairwise by projection-coefficient magnitude; spaces with at least as many wins as losses become trusted, yielding class-varying combinations.Equal-magnitude comparisons count as neither a win nor a loss.Source: §4.3 · pp. 7–8

4.4 Reliable Sample Filtering

  • Within each trusted space, label agreement retains examples whose observed class has the strictly highest discriminative-eigenvector alignment, then unions selected subsets.This relative-support criterion avoids classwise GMM fitting and its probability threshold; an example is retained if it passes in at least one trusted space for its observed class.Source: §4.4 · p. 8

5 Experiments

Across six synthetic-noise datasets and one realistic-noise dataset, REFINE achieves the highest average accuracy on every synthetic dataset and the highest mean accuracy on BSD. The experiments also examine fine-tuned encoders, selection reliability, trusted-space choices, and integration with other noisy-label training methods.

5.2 RQ1: Overall Effectiveness under Noisy Supervision

  • REFINE achieves the highest average accuracy on all six synthetic datasets and the highest mean accuracy on BSD; only FINE and REFINE outperform Standard on BSD.The synthetic results cover six noise settings per dataset; BSD is the realistic-noise dataset.Source: §5.2 · p. 10, Table 1 · p. 11, Table 2 · p. 12

Rakuten France

  • With jointly fine-tuned ResNet-50 and BERT encoders, REFINE achieves the highest accuracy under all four UPMC-Food101 noise settings.The encoders and classifier are jointly fine-tuned during training.Source: Table 3 · p. 12, Rakuten France · p. 12

5.3 RQ2: Selection Reliability and Mechanism Analysis

  • REFINE achieves the highest selection F1 across all four UPMC-Food101 settings; under P-IDN 50%, it retains 55.93% of hard-clean examples, and 91.88% of retained examples are supported by exactly one trusted space.Source: §5.3 · p. 12
  • Ablations show cumulative gains from LA, DE, and TSS, while trusted-space selection improves over ALL because no single space is consistently best.The most discriminative space combination also varies across classes; the reported comparisons support class-wise trusted-space selection.Source: §5.3 · pp. 12–14, Table 5 · p. 13, Table 6 · p. 14
  • Classification performance varies little as ϵ ranges from 10^-6 to 10^-2, although very small or large values weaken the intended regularization balance.An excessively small ϵ insufficiently regularizes the background matrix; an excessively large value increases the isotropic term's influence.Source: Figure 7 · p. 14, §5.3 · p. 14

5.4 RQ3: Generality across LNL Training Paradigms

  • Replacing Co-teaching's small-loss rule with REFINE improves accuracy under all settings, while REFINE combined with FreeMatch or DSS+ outperforms the respective original methods.R-DSS+ achieves the highest results under all settings.Source: §5.4 · pp. 14–15, Figure 8 · p. 15, Table 7 · p. 15
  • Adding REFINE increases accuracy for every tested training objective; GCE or JAL-CE combinations are slightly better than REFINE+CE under symmetric noise but lower under asymmetric and instance-dependent noise.How training objectives affect representation discriminability and selection quality across noise conditions remains for further investigation.Source: §5.4 · pp. 15–16, Figure 9 · p. 16

6 Conclusion

The conclusion presents REFINE as a class-wise method for selecting trusted fused and unimodal representation spaces and reliable examples under noisy supervision. Experiments across datasets, noise settings, and training paradigms show improvements in sample-selection quality and classification performance.

6 Conclusion

  • REFINE selects trusted representation spaces class-wise and identifies reliable examples through label agreement.It constructs discriminative eigenvectors by analyzing target and background classes.Source: §6 · p. 16
  • Experiments across diverse datasets, noise settings, and training paradigms show consistent improvements in sample-selection quality and classification performance.Source: §6 · p. 16
  • Future work will address partially corrupted, missing, and inconsistent modality-level signals within broader trustworthy multimodal learning.Source: §6 · p. 16

A.1 Proof of Theorem 1

The proof characterizes how FINE’s alignment score varies with class angle and contamination. Because all representation spaces share the same contamination ratio, angular behavior also determines their ordering.

A.1 Proof of Theorem 1

  • FINE strictly increases with θ_r across the full interval [0, π/2].Continuity extends the angular monotonicity to the entire interval.Source: §A.1 · pp. 18–19
  • For fixed θ_r in (0, π/2), FINE strictly decreases as τ increases over [0, 1).Source: §A.1 · p. 18
  • Since all spaces share τ for the same observed class, their FINE ordering follows their angular ordering.Source: §A.1 · pp. 18–19

A.2 Proof of Theorem 2

The proof derives discriminative directions from observed-class second-moment matrices and evaluates their alignment-score differences. It states that DE increases with class angle.

A.2 Proof of Theorem 2

  • DE strictly increases with θ_r over [0, π/2].Source: §A.2 · p. 19
  • The proof obtains DE from the leading generalized eigenvector of the observed-class second-moment matrices.It then uses the resulting direction to compute the alignment-score difference.Source: §A.2 · pp. 19–20
  • The proof normalizes the maximizing direction before substituting it into the alignment-score difference.Source: §A.2 · p. 20

A.4 Proof of Theorem 3

The proof establishes an ordering between representation spaces by combining a comparison of their projected quantities with angular monotonicity. It also bounds REFINE’s recall and precision and analyzes finite-sample estimation error.

A.4 Proof of Theorem 3

  • Combining the space comparison with angular monotonicity establishes the ordering in (41).The proof compares spaces r and s and reverses the ordering when the spaces are interchanged.Source: §A.4 · pp. 21–22

A.5 Lower Bounds on Recall and Precision

  • For fixed σ > 0 and τ > 0, the recall and precision lower bounds increase with sin²θ_r and with the alignment-score difference Δ_r.When τ = 0, precision and its lower bound equal one.Source: §A.5 · pp. 22–23

A.6 Finite-Sample Estimation Error

  • The finite-sample analysis bounds the estimation error of the class matrix and positive-class discriminative direction when N_min is sufficiently large and ϵ sufficiently small.Theorem 5 gives the bound with probability at least 1 − δ, conditional on the observed class counts.Source: §A.6 · p. 23

B.1 Datasets

The experiments use seven multimodal datasets, several noisy-label baselines, and four synthetic noise settings. Shared classifier and training configurations support comparisons across these settings.

B.1 Datasets

  • Experiments use seven multimodal datasets; cited descriptions identify UPMC-Food101, N24News, NWPU-Captions, Rakuten France, and VGGSound50, whose 50-class subset contains approximately 30,000 examples.Source: §B.1 · p. 25

B.2 Baselines

  • Baselines span supervised training, gradient- and training-dynamic-based selection, representation alignment, semi-supervised learning, and noise-tolerant losses.The sample-selection experiments integrate the mechanisms into a common multimodal classifier and train on retained examples with equal weights.Source: §B.2 · p. 26

B.3 Details of Label Noise Generation

  • Synthetic-noise experiments cover symmetric, asymmetric, full-modality instance-dependent, and partial-modality instance-dependent noise; compared methods share noisy labels for each setting and seed.The main settings use Sym at 50%, Asym at 40%, and F-IDN and P-IDN at 40% and 50%, with five random seeds per setting.Source: §B.3 · pp. 28–29

B.4 Hyperparameter Settings

  • The shared training setup uses a 30-epoch AdamW schedule, 256-dimensional modality and fused representations, and REFINE regularization parameter ϵ = 10^-4.For unfrozen encoders on UPMC-Food101, encoder and classification-head learning rates are 10^-6 and 10^-4, respectively.Source: §B.4 · p. 29

B.5 Details of Sample-Selection Experiments

The sample-selection experiments use shared classifier and training settings, with REFINE applied periodically after warm-up.

B.5 Details of Sample-Selection Experiments

  • REFINE uses all noisy examples for 10 warm-up epochs, then selects samples every five epochs during a 30-epoch training run.Methods are compared using the same multimodal classifier and shared training settings.Source: §B.5 · p. 30

B.6 Details of Semi-Supervised Experiments

REFINE is combined with semi-supervised methods by splitting selected and unselected examples into labeled and unlabeled subsets. The experiments also pair REFINE with noise-robust losses.

B.6 Details of Semi-Supervised Experiments

  • After warm-up, REFINE periodically assigns selected examples with observed labels to the labeled subset and the remaining inputs to the unlabeled subset for semi-supervised training.The paper integrates REFINE with FreeMatch and DSS+ and describes the FreeMatch combination as R-FreeMatch.Source: §B.6 · p. 30
  • Table 8 reports Macro-F1 across datasets and six noise settings, with best and second-best results indicated.Source: Table 8 · p. 31

Rakuten France

  • For the Rakuten France procedure, weak-view predictions generate pseudo-labels, and those passing adaptive thresholds supervise strong-view predictions alongside labeled-subset supervision and fairness regularization.The objective combines supervised loss, pseudo-label consistency loss, and FreeMatch’s self-adaptive class fairness regularization.Source: Rakuten France · pp. 31–32

B.7 Combination with Noise-Robust Loss Functions

  • GCE, SCE, and JAL-CE are each evaluated with and without REFINE, using the corresponding loss during warm-up and subsequent training on selected examples.Source: §B.7 · p. 32

C Additional Results

REFINE achieves the highest average Macro-F1 on all six datasets across six synthetic label-noise settings.

C Additional Results

  • REFINE achieves the highest average Macro-F1 on all six datasets under six synthetic label-noise settings.The experiments use the same setup as the main text; Table 8 reports the results.Source: C Additional Results · p. 32
Loading 2610.11057v1…