PaperlayerView exact revision on arXiv

Source-linked AI summary

Factorized Contrastive Learning: Going Beyond Multi-view Redundancy

Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis-Philippe Morency, Ruslan Salakhutdinov

arXiv:2306.05268v2cs.LGcs.AIcs.CLcs.CVcs.MM

TL;DR

Multimodal contrastive learning often assumes shared information is sufficient for downstream tasks, although task-relevant information can also be unique to one modality. FactorCL factorizes shared and unique information, uses mutual-information bounds and multimodal augmentations, and reports state-of-the-art results on six datasets.

  • Problem

    Multimodal contrastive learning assumes shared information between modalities is sufficient for downstream tasks, but real-world tasks can also depend on information unique to one modality.

    Source: §1 · p. 1

  • Method

    FactorCL factorizes task-relevant information into shared and unique representations, uses MI bounds to capture relevant and remove irrelevant information, and uses multimodal augmentations to approximate task relevance without labels.

    Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 6

  • Results

    FactorCL achieves state-of-the-art performance on six datasets, and on controlled data it maintains the best performance as unique information increases.

    Source: §1 · p. 2, §4.1 · p. 8

  • Takeaways & Limitations

    FactorCL extends contrastive learning to settings where downstream tasks depend on both shared and modality-unique information.

    Source: §1 · p. 2, §6 · p. 10

    The augmentation assumption holds when all information shared between an input and its augmentation is task-relevant.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data with only pairing information (e.g., image-caption or video-audio pairs). Underpinning these approaches is the assumption of multi-view redundancy - that shared information between modalities is necessary and sufficient for downstream tasks. However, in many real-world settings, task-relevant information is also contained in modality-unique regions: information that is only present in one modality but still relevant to the task. How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks? This paper proposes FactorCL, a new multimodal representation learning method to go beyond multi-view redundancy. FactorCL is built from three new contributions: (1) factorizing task-relevant information into shared and unique representations, (2) capturing task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bounds, and (3) multimodal data augmentations to approximate task relevance without labels. On large-scale real-world datasets, FactorCL captures both shared and unique information and achieves state-of-the-art results on six benchmarks

1 Introduction

Standard multimodal contrastive learning relies on shared information being sufficient for downstream tasks, but can miss task-relevant information unique to one modality. FactorCL factorizes shared and unique information and reports state-of-the-art results on six datasets.

1 Introduction

  • Standard multimodal contrastive learning can struggle when shared task information is scarce or when task-relevant information is unique to one modality.The introduction gives figurative captions, medical sensors, and robotic force sensors as examples.Source: §1 · p. 1
  • On controllable datasets, FactorCL maintains the best performance as unique information increases, while SimCLR and SupCon drop and Cross+Self recovers only in fully unique settings.Cross+Self performs less well at other shared/unique information ratios.Source: Figure 1 · p. 2
  • FactorCL achieves new state-of-the-art performance on six datasets spanning figurative language, affective tasks, and health prediction.The evaluated benchmarks include image-caption tasks and patient disease and mortality prediction from health indicators and sensor readings.Source: §1 · p. 2
  • FactorCL factorizes task-relevant information into shared and unique representations, capturing relevant information with MI lower bounds and removing irrelevant information with MI upper bounds.The method also uses multimodal augmentations to approximate task relevance without labels.Source: §1 · p. 2
  • Multimodal augmentations let FactorCL approximate task relevance without explicit labels in self-supervised learning.The paper presents this as its third key idea.Source: §1 · p. 2

2 Analysis of Multi-view Contrastive Learning

The analysis separates task-relevant shared and unique information from task-irrelevant shared information, then shows why standard contrastive learning can lose task-relevant information when modalities are non-redundant. The resulting uniqueness gap grows with task-relevant information unique to either modality.

2 Analysis of Multi-view Contrastive Learning

  • The analysis decomposes total task information into task-relevant shared information, task-irrelevant shared information, and unique information in each modality.It formalizes the setting with two modalities X1 and X2 and a task variable Y.Source: §2 · pp. 2–3
  • Multi-view redundancy assumes each modality contains at most a small amount of task-relevant information unavailable from the other, an assumption that excludes non-redundant cases.The paper cites health indicators, medical sensors, and robotic visual or force sensors as examples where unique information can matter.Source: §2 · p. 3
  • Under multi-view non-redundancy, standard contrastive learning has only a weak signal for shared information and struggles to learn task-relevant unique information.The analysis assumes optimal representations satisfy I(Z1;Y |X2) = I(Z2;Y |X1) = 0 in its suboptimality statement.Source: §2 · p. 3
  • The uniqueness gap measures task-relevant information lost in the encoded representation and increases as modality-unique task information grows.The analysis also states that encoded task information falls with lower shared information or higher task-irrelevant shared information, and that downstream error rises with higher unique and lower shared information.Source: §2 · p. 3

3 FACTORIZED CONTRASTIVE LEARNING

FACTORCL extends contrastive learning by representing task-relevant shared and modality-unique information separately. It uses contrastive mutual-information bounds and multimodal augmentations to learn those factors without labels.

3 FACTORIZED CONTRASTIVE LEARNING

  • FACTORCL learns separate shared and modality-unique representations for task-relevant information in the two inputs.The framework recovers standard contrastive learning when unique information is zero or all shared information is task-relevant.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 Supervised FACTORCL with shared and unique information

  • INCE lower bounds and INCE-CLUB upper bounds mutual information, while minimizing a conditional upper bound removes task-irrelevant shared information.INCE-CLUB reuses INCE's optimal critic, avoiding separate optimization of the upper bound.Source: §3.1 · p. 5

3.2 Self-supervised FACTORCL via multimodal augmentations

  • Multimodal augmentations replace labels as proxies for task relevance, with unique augmentation preserving information in one modality that the other already indicates.For example, image augmentation should avoid cropping caption-referenced content; flipping or color jittering are offered as alternatives.Source: §3.2 · p. 6, Figure 4 · p. 7
  • Separate unimodal augmentations can work well but do not necessarily satisfy the conditional-mutual-information equality required for exact substitution.The paper defines optimal multimodal augmentations to satisfy that property.Source: §3.2 · p. 6

3.3 Overall method and implementation

  • Algorithm 2 takes multimodal data as input, and learned representations can feed linear classifiers for classification or retrieval.The method uses trainable neural networks to extract representations from the modalities.Source: §3 · p. 4, §3.3 · p. 7

4 Experiments

Experiments test FactorCL on controlled synthetic data and real-world multimodal benchmarks with varied shared and unique task information. FactorCL outperforms comparison methods on real-world tasks, while probes and ablations examine what its representations capture and which components matter.

4. Experiments

  • The authors state that, under tight MI-bound estimates, FACTORCL’s representations recover task-relevant shared and unique information from both modalities.In practice, the authors do not expect perfect MI estimation or maximization.Source: §4 · p. 7
  • The synthetic setup varies task-relevant shared and unique information by generating labels from latent factors shared across modalities or unique to either modality.Real-world evaluation spans six MultiBench datasets, seven modalities, and more than 84,000 data points, covering healthcare, affective computing, and multimedia.Source: §4.1 · p. 8, §4.2 · p. 8

4.1 Controlled experiments on synthetic datasets

  • On synthetic data, FACTORCL maintains the best performance as unique information increases; Cross+Self recovers only at the fully unique setting, while SimCLR and SupCon decline.Probing finds 7 bits of unique and 10 bits of shared information for FACTORCL; its factorized representations align shared information with ZS and modality-specific information with U1 and U2.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 Self-supervised multimodal learning with low redundancy and high uniqueness

  • On MultiBench, FACTORCL significantly outperforms baselines lacking both shared and unique representations in supervised and self-supervised settings, particularly on MUSTARD and MIMIC.Table 2 reports results across datasets with varying shared and unique information.Source: Table 2 · p. 9, §4.2 · p. 9
  • Without factorization, performance drops by 6.1% on average and by as much as 8.6% on MIMIC; without the upper-bound objective, drops reach 13.6% on average and 23.5% on MOSI.On IRFL, joint multimodal augmentations achieve 95% versus 92% for separate unimodal augmentations, and both outperform SimCLR and Cross+Self.Source: §4.2 · p. 9
  • Using only shared or unique representations lowers performance versus full FACTORCL; on MUSTARD, U1-only reaches 59.4% against 69.9% for full multimodal prediction.MOSEI and MOSI favor shared and U2-only models, while MUSTARD favors U1-only prediction.Source: §4.2 · p. 9, Table 4 · p. 10
  • On IRFL, FACTORCL substantially improves results over zero-shot and fine-tuned CLIP and continued-pretraining baselines for classifying images with figurative captions.Table 3 covers continued pre-training on CLIP for image and figurative-language classification.Source: Table 3 · p. 9, §4.2 · p. 9

5 Related Work

Related work connects FactorCL to multimodal and contrastive learning, factorized representations, and information-theoretic analyses of multimodal data. The paper distinguishes its focus on task-relevant shared and unique information from these neighboring approaches.

5 Related Work

  • Contrastive learning spans vision [11] [13] [14] [28] [57], language [24] [53] [55], speech [5] [57] [62], and multimodal tasks [1] [36] [60].Prior work addressing multi-view redundancy limitations uses autoencoder reconstruction or unimodal contrastive learning [38].Source: §5 · p. 10
  • Multimodal contrastive learning aligns paired data across modalities, with methods optimizing inter-modal pairs or both intra- and inter-modal pairs [1] [2] [36] [60] [33] [35] [41] [43].The paper also relates FactorCL to factorized representations that capture modality-specific and redundant multimodal information [32] [74].Source: §5 · p. 10
  • Information-theoretic work in multimodal learning studies co-learning [61] and multi-view learning [34] [75], alongside practical lower and upper bounds [57] [58] [59] [83].FactorCL builds on an upper bound from Cheng et al. [16] and relates its shared/unique information definitions to partial information decomposition [81], co-information [79], and interaction information.Source: §5 · p. 10

6 Conclusion

The conclusion identifies task-relevant modality-unique information as a challenge for standard contrastive learning and presents FACTORCL as a factorized approach. It points to masked and non-contrastive pre-training as future directions.

6 Conclusion

  • Standard contrastive learning can suffer when task-relevant information lies in modality-unique regions, a situation the paper says is common in real-world applications.Examples include sensor placement, medical testing, and multimodal interaction.Source: §6 · p. 10
  • FACTORCL combines factorized representations, MI upper bounds to remove task-irrelevant information, and multimodal augmentations to approximate the unobserved task.Source: §6 · p. 10
  • The authors identify extending these ideas to masked and non-contrastive pre-training as future research directions.Source: §6 · p. 10

A Broader Impact

The paper frames FactorCL as a way to broaden self-supervised contrastive learning beyond redundant views, while emphasizing careful deployment of multimodal models.

Broader applicability

  • FactorCL is intended to broaden self-supervised contrastive learning to multimodal tasks beyond those exhibiting multi-view redundancy.The authors say special care is needed to ensure these models are safely deployed for real-world benefit.Source: A Broader Impact · p. 16

Time and space complexity

  • The authors argue that FACTORCL does not significantly increase complexity relative to heuristic combinations of cross-modal and single-modality contrastive learning.They cite reuse of the INCE optimal critic, shared encoders, and adequate performance from separate unimodal augmentations.Source: A Broader Impact · p. 16

Privacy and security

  • The authors acknowledge privacy risks from predictions on recorded human behavior and medical data, and restrict use of these datasets to scientific study.They report anonymization and de-identification measures, including using pre-extracted non-invertible affect-recognition features.Source: A Broader Impact · p. 16

Social biases

  • Imbalanced datasets may create social biases involving gender, race, and ethnicity; adapting conditional contrastive learning to sensitive attributes is proposed for future study.The authors identify this as an important direction for future work.Source: A Broader Impact · p. 16

Future work

  • The authors identify higher-dimensional and complex inputs, manual augmentation selection, and task-specific weighting of shared versus unique information as areas for future work.They suggest better MI bounds, automatically generated augmentations, and reweighting objective terms.Source: A Broader Impact · p. 16

B Analysis of Multi-view Contrastive Learning

This section explains why standard multimodal contrastive learning can miss task-relevant unique information, then develops contrastive estimators and augmentation strategies for shared and conditional information.

B Analysis of Multi-view Contrastive Learning

  • Under multi-view non-redundancy, standard contrastive representations can be suboptimal because they omit task-relevant information unique to either modality.The section distinguishes this condition from redundancy, where unique task-relevant information is bounded by ε, and reports empirical support for non-redundancy.Source: B Analysis of Multi-view Contrastive… · p. 17

C.1 Contrastive estimators

  • INCE supplies a lower bound on shared mutual information, while INCE-CLUB uses INCE’s optimal critic to provide an upper bound.The plug-in upper bound avoids separately optimizing a second critic.Source: §C.1 · p. 19

C.2 Unimodal and multimodal augmentations

  • Task-relevant augmentations should preserve relevant information while changing irrelevant information; multimodal augmentation proceeds through unimodal and then unique augmentation.The example given for image classification treats the object as relevant and the background as irrelevant.Source: §C.2 · pp. 19–20

C.2.1 Implementing conditional CL via kernel

  • Kernel weighting estimates conditional contrastive objectives without directly sampling conditional distributions or training a generative model.The method uses similarities between conditioned augmented pairs to weight samples.Source: §C.2.1 · pp. 20–21

C.3 Final estimators in FACTORCL

The final estimators separate shared and modality-unique task information; under stated ideal conditions, their representations recover the total task-relevant information. The section also sketches extensions to non-contrastive and masked pretraining.

C.3 Final estimators in FACTORCL

  • Under the specified augmentation assumptions, INCE lower bounds and INCE-CLUB upper bounds are substituted into the shared and unique information objectives.The passage states that these estimators bound both unconditional and conditional mutual information.Source: §C.3 · p. 21
  • With perfectly maximized objectives and tight estimators, FACTORCL’s shared and unique representations sum to the input modalities’ total task-relevant information.The decomposition combines shared information with each modality’s task-relevant information conditional on the other representation.Source: §C.3 · p. 21

C.4 Extensions to masking and non-contrastive learning

  • For non-contrastive learning, the authors propose regularizing unimodal and cross-modal features, conditioning augmentation to isolate unique features.They also sketch conditional masking so that masked prediction captures unique information, while noting these extensions need further study.Source: §C.4 · pp. 22–23

D.1 Implementation details

The implementation uses separate encoder heads and critics for the contrastive objectives, with conditioning introduced by concatenation. It approximates conditional sampling with augmented pairs and trains the critics using an inner update.

Architecture

  • The implementation adds two MLP heads to the encoders and uses a separate critic for each objective, with concatenation as the critic design.Each critic receives the outputs of the relevant heads.Source: Objective Formulation and Architecture · p. 23

FACTORCL-SUP and FACTORCL-SSL

  • The supervised variant conditions objectives on labels, while the self-supervised variant uses modality augmentations for its conditional terms.The supervised representation heads are combined according to whether they serve shared or unique objectives.Source: Objective Formulation and Architecture · pp. 23–24

Conditioning by concatenation

Estimation of CMI

  • The conditional estimator approximates conditional samples with augmented pairs because direct conditional sampling can be expensive.The submission uses one additional augmented pair for consistency, and reports empirical comparability to existing CMI estimators and the expected lower-bound ≤ true CMI ≤ upper-bound ordering.Source: Objective Formulation and Architecture · p. 24

Training Strategy

  • Using one inner-loop critic update worked well and kept training time comparable to existing contrastive methods.The authors report that setting k = 1 did not greatly affect convergence and still produced promising results.Source: Objective Formulation and Architecture · p. 25

D.2 Datasets

The experiments span synthetic data with controlled information structure and real-world multimodal tasks, including healthcare, sentiment, humor, sarcasm, and figurative-language classification. Additional image-classification experiments use CIFAR10 and MNIST.

Gaussian datasets for MI estimation

  • Four multivariate-Gaussian datasets use dimensions {20, 50, 100, 200} and ground-truth MI values {2, 4, 6, 8, 10}; each setting is sampled 4,000 times with batch size 64.These datasets are used to assess MI estimation.Source: §D.2 · p. 25

Synthetic dataset with controlled generation

  • Controlled synthetic data varies the task-relevant contributions of shared and modality-unique latent vectors, which are transformed into high-dimensional modality inputs.Labels depend nonlinearly and noisily on varying ratios of the shared and unique vectors.Source: §D.2 · p. 25

Multimodal fusion datasets

  • The real-world evaluation uses five MultiBench fusion datasets plus IRFL, with tables covering MultiBench results and continued CLIP pretraining for figurative-language classification.The MultiBench collection and IRFL test settings with varying amounts of shared and unique task-relevant information.Source: §D.2 · p. 25, Table 6 · p. 26, Table 7 · p. 26
  • MIMIC-III tests binary ICD-9 group 7 classification from hourly time series and static patient data, while MOSEI and MOSI test binary sentiment classification using vision and text features.MOSEI and MOSI sentiment intensities are converted to positive-versus-negative labels.Source: §D.2 · pp. 25–26
  • UR-FUNNY and MUSTARD provide humor and sarcasm tasks, while IRFL pairs images with figurative captions labeled as idiom, simile, or metaphor.UR-FUNNY uses laughter-marked punchlines and sampled negative examples; MUSTARD uses audiovisual utterances with textual context.Source: §D.2 · pp. 26–27

Additional image-classification experiments

  • Additional CIFAR10 [42] and MNIST [19] experiments apply FACTORCL objectives to image classification; a described setup continued pretraining pretrained CLIP [60] models using CLIP-VIT-B/32 rather than training from scratch.The pretrained image and text encoders were trained for 10 epochs with Adam at a 1e-6 learning rate.Source: Table 8 · p. 27, §D.2 · p. 27

D.3 Additional analysis and results

FactorCL is evaluated across multimodal fusion benchmarks, figurative-language classification, and additional analyses of high-shared-information settings and conditional mutual-information estimators. The results report benchmark gains and evidence that the conditional estimators behave as lower and upper bounds.

Fusion experiments

  • Across five independent runs, FACTORCL significantly outperformed baselines lacking both shared and unique information in supervised and self-supervised settings, particularly on MUSTARD and MIMIC; MOSEI and MOSI improved 6.8% and 21.9%, respectively, versus SupCon.Source: §D.3 · p. 27

IRFL results

  • On IRFL, FACTORCL outperforms zero-shot and fine-tuned CLIP and continued-pretraining baselines; its self-supervised unique augmentations also beat independent augmentations.The supervised version gives the best overall results; the text augmentation uses word masking, and the unique image augmentation removes cropping.Source: §D.3 · pp. 27–28

Empirical verification on InfoMin assumption

  • The InfoMin probe reports I(X1;X2) = 12.29 and I(Z1;Y|X2) = 0.4, with the latter much smaller and closer to zero, supporting the assumption's reasonableness.The authors use this comparison as evidence that Theorem 1 holds in practice.Source: Table 11 · p. 28, §D.3 · p. 28

Additional experiments on high shared information and low unique information

  • On CIFAR10 and MNIST, FactorCL outperforms self-supervised contrastive learning and roughly matches supervised contrastive learning in high-shared, low-unique settings.The authors conclude that FactorCL recovers the performance of methods that do not capture unique information when shared information is abundant.Source: §D.3 · p. 28

Experiments on CMI estimator verification

  • Conditional InfoNCE estimates fall below true conditional MI, while Conditional InfoNCE-CLUB estimates exceed it, with performance comparable to estimators in [54].The verification uses synthetic data varying sample count or representation dimension.Source: §D.3 · p. 28, Table 10 · p. 28
Loading 2306.05268v2…