PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions

Anil Rahate, Rahee Walambe, Sheela Ramanna, Ketan Kotecha

arXiv:2107.13782v3cs.LGcs.AI

TL;DR

Real-world multimodal systems face missing, noisy, weakly annotated, and scarce modalities, while dedicated co-learning reviews remain lacking. This survey organizes implementations, techniques, applications, datasets, open problems, and future directions through a comprehensive multimodal co-learning taxonomy.

  • Problem

    Multimodal co-learning addresses the gap between idealized systems with complete, aligned, noiseless modalities and real-world settings where modalities may be missing, noisy, scarce, or weakly annotated.

  • Method

    The survey reviews implementations and techniques across co-learning challenges and organizes them into a taxonomy covering modality presence, data parallelism, noise, annotations, domain adaptation, interpretability, and fairness.

  • Results

    The survey consolidates studies by co-learning objective and identifies which objectives are widely addressed versus requiring further focus.

  • Takeaways & Limitations

    Key future directions include supporting missing modalities during training and testing, measuring co-learning, handling non-parallel and noisy data, reducing labeled-data dependence, and expanding multimodal datasets.

  • Takeaways & Limitations

    Attention-based emotion recognition using aligned transcript and audio has not been experimentally evaluated for missing-modality performance, and multimodal models lack uniform robustness practices across noisy conditions and applications.

Abstract

from arXiv · show

Multimodal deep learning systems which employ multiple modalities like text, image, audio, video, etc., are showing better performance in comparison with individual modalities (i.e., unimodal) systems. Multimodal machine learning involves multiple aspects: representation, translation, alignment, fusion, and co-learning. In the current state of multimodal machine learning, the assumptions are that all modalities are present, aligned, and noiseless during training and testing time. However, in real-world tasks, typically, it is observed that one or more modalities are missing, noisy, lacking annotated data, have unreliable labels, and are scarce in training or testing and or both. This challenge is addressed by a learning paradigm called multimodal co-learning. The modeling of a (resource-poor) modality is aided by exploiting knowledge from another (resource-rich) modality using transfer of knowledge between modalities, including their representations and predictive models. Co-learning being an emerging area, there are no dedicated reviews explicitly focusing on all challenges addressed by co-learning. To that end, in this work, we provide a comprehensive survey on the emerging area of multimodal co-learning that has not been explored in its entirety yet. We review implementations that overcome one or more co-learning challenges without explicitly considering them as co-learning challenges. We present the comprehensive taxonomy of multimodal co-learning based on the challenges addressed by co-learning and associated implementations. The various techniques employed to include the latest ones are reviewed along with some of the applications and datasets. Our final goal is to discuss challenges and perspectives along with the important ideas and directions for future work that we hope to be beneficial for the entire research community focusing on this exciting domain.

1. Introduction

Multimodal co-learning addresses real-world conditions in which modalities may be missing, noisy, weakly annotated, or unavailable, while existing multimodal learning taxonomies do not fully cover these challenges. This survey expands the taxonomy and reviews implementations, applications, datasets, challenges, and future directions.

  • Multimodal learning combines modalities such as language, vision, audio, physiological signals, and physical sensors to use complementary information.
  • Real-world multimodal systems must handle modalities that are missing, noisy, limited, or poorly annotated during training or testing.
  • Multimodal co-learning transfers knowledge from an informative modality to a less informative modality, often when the supporting modality is unavailable at inference.
  • The survey expands earlier co-learning classifications into a taxonomy based on modality conditions, noise, missing modalities, annotations, pairing, and domain differences.
  • It organizes literature on co-learning and related representation, alignment, translation, and fusion methods, alongside datasets, applications, open problems, and future directions.

2. Research goals and methodology

The study formalizes multimodal co-learning objectives and systematically examines research that addresses them, including work not explicitly labeled as co-learning. It uses research questions to organize objectives, taxonomies, and historical comparisons.

  • The study defines objectives for conditions where one or more modalities are fully or partly missing during training or testing.
  • Because few papers explicitly discuss co-learning, the authors systematically searched studies addressing one or more co-learning objectives without necessarily using that terminology.
  • The research questions are presented as a framework for examining multimodal co-learning objectives and associated implementations.
  • The methodology also includes single-modality studies to provide comparison and historical perspective on techniques supporting co-learning objectives.

3. Historical perspective to multimodal co-learning

Historical work spans multi-view and semi-supervised learning, noisy-label handling, and teacher-student methods that support multimodal co-learning objectives under limited or imperfect data conditions.

  • Multi-view and semi-supervised learning: Multi-view learning optimizes separate functions for heterogeneous data views with different statistical properties.The paper presents multi-view learning as a response to single-view models' inability to handle heterogeneous data.
  • Noisy-label learning: 45%-20% noisy labels: Co-teaching showed better performance across high-to-low noise levels, but peer teaching allowed error flow between networks.Each network selected small-loss instances and used them to train its peer.
  • Noisy-label learning: Co-teaching+ maintains network divergence through disagreement-based data and parameter updates, while JoCoR uses supervised and co-regularization losses.The comparison describes distinct error-flow patterns across MentorNet, Decoupling, Co-teaching, Co-teaching+, and JoCoR.
  • Teacher-student learning: Self-training can amplify prediction errors, motivating teacher-student networks for selecting pseudo-labels.Teacher-student networks were initially introduced for distilling a large teacher into a smaller, faster student.
  • Recent extensions: Recent teacher-student extensions add fine-tuning, improved data selection, perturbation robustness, or language-model augmentation.Reported applications include image and video classification using 1 billion unlabeled images and task-specific in-domain text extraction.
  • Multi-view and semi-supervised learning: Co-training uses independent views and proxy labels to add predicted labels to unlabeled data.Its seminal application combined labeled and unlabeled web-page data using content and link views.
  • Multimodal extensions: Multimodal co-learning adapts these objectives by letting one modality assist another during training even when it is absent at testing.Teacher-student networks and self-training are presented as mechanisms for semi-supervised or weakly supervised settings.

4. Multimodal co-learning taxonomy

The proposed taxonomy classifies multimodal co-learning by modality conditions at training and testing, covering noise, missingness, annotation availability, pairing, and cross-domain data.

  • Co-learning setup: In multimodal co-learning, a supporting modality transfers knowledge to another modality during training and is usually absent at inference.The same support relationship can extend beyond two modalities.
  • Taxonomy basis: The taxonomy is based on modality conditions at training and test time, including noise type, missing modalities, annotation availability, pairing, and dataset or domain differences.It also incorporates data parallelism from the broader multimodal machine-learning taxonomy.
  • Survey organization: The survey organizes each co-learning objective by definition, classification, studies, methods, outcomes, data strategy, and applications.Multimodal sentiment classification using audio, video, and text illustrates selected concepts.

4.1 Presence of modality

Multimodal co-learning classifies modality availability across training and testing, including fully or partly missing modalities and the direction of support between dominant and weaker modalities. The surveyed approaches use reconstruction, distillation, attention, transfer, and related methods to address these conditions, although training-time missing modalities remain challenging.

  • Dominant modalities: The dominant modality at test time motivates stronger-enhancing-weaker and weaker-enhancing-stronger categories, although these are not explicit taxonomy subclasses.The distinction identifies which modality supports the modality used for testing.
  • Fully missing modality: Late fusion can tolerate an absent test modality but treats modalities independently and therefore cannot learn multimodal interactions.This limitation makes late fusion less suitable for co-learning.
  • Fully missing modality: Fully missing test modalities can be handled through co-training, knowledge distillation, hallucination networks, autoencoder reconstruction, and multimodal factorization.Examples include teacher-student transfer from RGB to depth, DCC-CAE reconstruction, and factorized discriminative and generative representations.
  • Partly missing modality: Partly missing modalities cover temporary or incomplete sensor absence, such as brief camera failure or unavailable audio, requiring models to maintain performance across varying missing-data percentages.The missing proportion may range from mild to severe.
  • Presence of modality: Modality presence is classified by whether modalities are available fully or partly during training and testing, including cases with multiple modalities at training but one at testing.The taxonomy considers missing modalities at both training and testing, as well as partial absence.
  • Discussion: Handling missing modalities depends on the amount of co-learning; autoencoder and factorization methods outperform unimodal extensions, while training-time absence remains challenging.GAN- and meta-learning-based frameworks are described as a step forward for missing modalities during training.

4.2 Data parallelism

Multimodal co-learning classifies data relationships as parallel, non-parallel, or hybrid, depending on how modalities are paired or linked. Recent techniques increasingly use weak or bridged relationships because strong pairing is costly and time-consuming.

  • Parallel data: Parallel data directly aligns modality samples, such as word-level text–audio pairs, and supports knowledge transfer between modalities.Image-caption pairs and synchronized audio, video, and language segments are examples used in multimodal applications.
  • Parallel data: Strongly paired data is intuitive for multimodal co-learning, but preparing it becomes inefficient as the number of modalities increases.Offline preprocessing can restrict end-to-end learning, while finer-level alignment may be unavailable even when coarse alignment exists.
  • Non-parallel data: Attention mechanisms can align transcripts with audio without manual pairing, although their performance under missing-modality co-learning remains untested.The cited ASR emotion-recognition implementation aligns text and audio automatically rather than constructing strong pairs.
  • Non-parallel data: Non-parallel data uses shared concepts rather than direct observation-level links, enabling methods such as DeViSE and video–text joint embedding.Long-sequence techniques and MIL-NCE exploit coarse instructional-video alignment without strong video–text pairing.
  • Hybrid data: Hybrid data creates modality pairings through a shared modality or dataset, often using a pivot such as images or video.Bridge relations support aligned multimodal and multilingual embeddings, and video–multilingual-text transformers outperform video–English-text transformers for multilingual video search and zero-shot learning.
  • Summary: The survey identifies parallel, non-parallel, and hybrid data as three data-parallelism types supported by contemporary multimodal co-learning methods.Recent techniques use non-parallel and hybrid data instead of always requiring strongly paired data.

4.3 Noisy Modality

Multimodal co-learning addresses label noise, sample noise, and class or modality imbalance, extending robustness techniques across modalities. Adversarial methods, generative models, and multimodal fusion are prominent approaches, but robustness remains limited in some settings.

  • Noise types: Noisy conditions include label noise and data-sample noise, motivating teacher–student, co-sampling, coteaching, adversarial, and generative approaches.Label noise can arise from annotator errors, weak expertise, complex or subjective annotations, and poor-quality samples.
  • Data noise: Multimodal models use supplementary information across modalities to improve robustness against noisy inputs.The survey discusses noisy physiological signals and multimodal techniques that also support missing-modality objectives.
  • Data noise: Late fusion proved more robust than early and hybrid fusion under audio perturbations in audio-visual event recognition.Low-frequency adversarial noise had higher attack potential, while classes trained on more data were more robust in an imbalanced dataset.
  • Data noise: Pre-trained vision-language models are more robust than task-specific state-of-the-art models, but their robustness remains limited.MANGO generates embedding-space noise and combines image masking with token dropping across four robustness settings and nine datasets.
  • Data noise: Adversarial noise generation has expanded from images to text, audio, EEG, and physiological data, with modality-specific noise conditions.The survey identifies adversarial techniques as leading approaches for multimodal noisy-data research.
  • Data imbalance: Class imbalance can reduce performance, overrepresent majority classes, and cause minority classes to be ignored or treated as noise.Oversampling, undersampling, SMOTE, ADASYN, GANs, and multimodal feature generation are reviewed as balancing strategies.

4.4 Modality annotations

Multimodal co-learning reduces reliance on costly, error-prone annotations through semi-supervised, weakly supervised, and unsupervised techniques. These methods exploit proxy labels, semantic relationships, high-level supervision, and unsupervised alignment across modalities.

  • Motivation: Multimodal annotation is costly because modalities may require separate experts, labels, and alignment, while labels can also be noisy or error-prone.Real-world multimodal systems therefore explore unsupervised, semi-supervised, and weakly supervised learning.
  • Supervised learning: Supervised multimodal models can support missing- or noisy-modality objectives, but they require labeled and often strongly paired data.Examples include sentiment analysis with paired audio, video, and language, and vision-language tasks tested with one modality.
  • Semi-supervised learning: Semi-supervised learning combines a small annotated set with larger unannotated data, including cases where only one modality is labeled.Proxy labels and supporting modalities can transfer supervision across image–tag or image–text data.
  • Weak supervision: Weak supervision uses higher-level relationships to perform lower-level tasks, such as mapping image regions to caption phrases or using video metadata for classification.Contrastive learning and multimodal co-training provide examples of this strategy.
  • Unsupervised learning: Unsupervised multimodal methods create mappings or alignments from unannotated data, including object–caption correspondences and recipe text–video relationships.Unsupervised alignment can support co-learning because modality alignment is crucial for knowledge transfer.
  • Discussion: The survey concludes that semantic embedding, pre-trained models, inherent cross-modal relationships, and contrastive alignment can enable co-learning with limited or absent labels.The reviewed studies span representation, alignment, fusion, and co-learning objectives across multiple modality conditions.

4.5 Multimodal domain adaptation

Multimodal domain adaptation addresses distribution shifts, differing modality availability, and missing target modalities across homogeneous, heterogeneous, and cross-modal settings. Reviewed solutions use domain-invariant representations, auxiliary data, transfer learning, embedding spaces, and GANs.

  • Definitions: Domain adaptation transfers knowledge from labeled source data to an unknown target domain, while domain generalization uses multiple labeled source domains without target samples.Both frameworks are extended to multimodal settings with modality and label-space variations.
  • Homogeneous adaptation: Homogeneous multimodal adaptation retains the modalities across source and target domains but may face different recording or noise conditions.Examples include audio-video data collected in controlled versus real-world settings and image-text data from different sources.
  • Adaptation mechanisms: Multimodal adaptation learns domain-invariant features using intra-modality, inter-modality, and inter-domain properties, often with adversarial networks.The objective is to prevent fused output features from discriminating between source and target domains.
  • Heterogeneous adaptation: Heterogeneous multimodal adaptation handles different source and target modalities or missing target modalities using auxiliary complete-modality databases and cross-modality transfer.A latent low-rank constraint supports transfer both across modalities and across databases.
  • Heterogeneous adaptation: Heterogeneous multimodal adaptation can use auxiliary datasets, co-occurrence-based embedding spaces, transfer learning, or GANs.These approaches reconstruct or generate target-modality representations when source and target feature spaces differ.
  • Cross-modal generalization: Cross-modal generalization aligns knowledge from one source modality to another target modality without requiring target-domain samples, except potentially a few few-shot examples.Cross-modal meta-alignment and weak supervision support applications such as image-to-audio transfer and medical image labeling.
  • Discussion: The survey presents multimodal domain adaptation as a route toward co-learning under shifted distributions and missing target modalities, while noting that its definitions remain evolving.The reviewed framework combines intra-modality, inter-modality, and inter-domain attributes with auxiliary databases for knowledge transfer.

4.6 Interpretability and fairness

The survey connects multimodal co-learning with interpretability, explainability, fairness, and bias objectives, especially under missing or noisy modality conditions. It reviews factorization, disentanglement, GAN, counterfactual, and contrastive approaches.

  • Objectives: Multimodal co-learning is framed as supporting interpretability, explainability, fairness, and bias objectives alongside handling missing or noisy modalities.The paper asks whether multimodality during training can improve interpretability and fairness when only one modality is available at test time.
  • Interpretability and explainability: Multimodal models can expose modality contributions and shared versus modality-specific factors through factorization and disentanglement.MFM separates discriminative factors common across modalities from generative factors specific to each modality.
  • Interpretability and explainability: GAN variants support co-learning by generating absent modalities, handling noisy labels or conditions, and producing interpretable representations.The reviewed examples include CMCGAN, rGAN, cGAN, AC-GAN, dacssGAN, and InfoGAN.
  • Interpretability and explainability: Counterfactual analyses test modality bias by removing or distorting inputs, while co-learning methods inherently accommodate missing and noisy conditions.The survey describes examining language bias in VQA when text is absent and visual bias when visual input is absent or distorted.
  • Fairness and bias: Multimodal co-learning is presented as a route to interpretability and explainability through factorization, disentanglement, GAN variants, counterfactual explanations, and contrastive learning.The section also notes that bias can arise from annotators, imbalanced data, feature selection, and other sources.

4.7 Summary of multimodal co-learning taxonomy

The survey consolidates studies by co-learning objective and associated multimodal challenge areas. This organization reveals which objectives are widely addressed and which require further research.

  • Summary: The consolidated review includes studies achieving one or more multimodal co-learning objectives, including work without missing modalities at test time.Table 8 links the reviewed studies to corresponding multimodal challenge areas.
  • Summary: The taxonomy-based organization helps identify widely addressed co-learning objectives and objectives needing further focus.The survey positions this comparison as a basis for future research directions.

5. Computational approaches for multimodal co-learning

The paper reviews computational approaches for multimodal co-learning across fusion, attention, encoder-decoder, generative, multi-task, transfer, few-shot, and meta-learning methods. These methods address missing, noisy, weakly annotated, or scarce modalities through complementary mechanisms.

  • Overview: Multimodal co-learning implementations combine representation, alignment, translation, fusion, and co-learning techniques rather than treating these challenges independently.The paper describes computational approaches as supporting multiple co-learning objectives.
  • Fusion: Fusion methods include tensor, low-rank, hierarchical, and architecture-search approaches, with optimization adapted to heterogeneous modalities.The overfitting-to-generalization ratio is used to control loss weighting during training.
  • Fusion: Late fusion manages missing modalities at test time, while hybrid fusion, multi-task learning, memory networks, and attention address missing or noisy conditions.A memory fusion network models view-specific and cross-view-specific interactions; training can use audio, video, and text while testing uses text only.
  • Attention and transformers: Attention, co-attention, and transformers support cross-modal alignment, feature selection, missing-modality handling, noisy conditions, data parallelism, and unsupervised annotations.Co-attention maps question words to image objects, while multimodal transformers encode separate modalities jointly.
  • Other approaches: Autoencoders, GAN variants, and multi-task learning support co-learning through missing-data handling, noise robustness, modality generation, domain adaptation, and shared representations.Transfer, few-shot, and meta-learning move knowledge from data-rich or clean modalities to scarce, noisy, or absent modalities.
  • Transfer and few-shot learning: Transfer learning, multimodal embeddings, pre-trained models, and few-shot learning support missing-modality, noisy-modality, data-parallelism, and annotation objectives.The surveyed methods transfer knowledge from high-resource or clean modalities to low-resource or noisy modalities.

5.9 Meta-Learning

The section places meta-learning, multi-instance learning, self-supervised learning, contrastive learning, and domain adaptation within multimodal co-learning. These approaches target missing or noisy modalities, weak annotation, cross-modal transfer, and generalization.

  • Meta-Learning: Meta-learning improves data efficiency, enables knowledge transfer, and supports unsupervised learning across task-agnostic and task-specific settings.It is categorized as metric-based, model-based, or optimization-based.
  • Meta-Learning: MAML has been extended to reconstruct severely missing modalities, while CROMA aligns representations across source and target modalities.These methods address missing modalities during training and testing and support cross-modal generalization.
  • Multi-instance learning: Multi-instance learning enables weakly supervised multimodal learning without explicit data alignment, reducing labeling requirements for large datasets.Applications include sound classification, image-caption object classification, and violent-video detection.
  • Contrastive learning: Multimodal contrastive learning can outperform unimodal learning with suitable losses and supports cross-modal representation alignment.Reviewed applications include multimodal datasets, phrase grounding, VQA pretraining, and weakly supervised learning from web data.
  • Contrastive learning: CVLP outperformed ViLBERT, LXMERT, and other pre-trained vision-language models on VQA datasets using contrastive loss on the visual branch.Contrastive methods also improved weakly supervised phrase grounding by contrasting related and non-related regions.
  • Domain adaptation: Multimodal domain adaptation and generalization address missing modalities, noisy conditions, limited annotations, and cross-domain knowledge transfer.Examples include auxiliary modality databases, embedding spaces, and domain translator functions.

6. Datasets and Applications of multimodal co-learning

Multimodal co-learning applications span diverse tasks and modality combinations, with representative studies pairing methods and datasets to address practical multimodal objectives.

  • Applications cover classification, regression, recognition, detection, sentiment and emotion analysis, speech enhancement, sound classification, remote sensing, rumor detection, deception detection, recruitment, and vision-language tasks.
  • Table 11 organizes representative studies by application category, modalities, methods, and corresponding datasets.
  • Representative co-learning objectives include reconstructing missing modalities, using text as a source modality for domain adaptation, and generating missing modality data.
  • Datasets span image-text, image-audio, text-speech, video-text, and multimodal video settings, including Yummly-28K, CIFAR, ESC50, Wilderness, and FCVID.
  • The surveyed implementations use methods including meta-alignment with Noise Contrastive Estimation, multimodal transformers, CycleGAN, bridge correlation networks, and co-training.

7. Open problems and future research directions

The survey identifies open problems in multimodal co-learning involving representations, datasets, evaluation, alignment, noise, annotation, domain adaptation, bias, and fairness. It proposes broader support for missing modalities and better diagnostics to measure co-learning across conditions.

  • Representation: Universal multimodal representations must preserve complementary and supplementary information while retaining intra-modality structure and supporting configurable modality sets.
  • Pre-trained models: Audio and video face a shortage of large datasets and pre-trained models, increasing reliance on language modalities or risking overfitting.
  • Datasets: Large, realistic multimodal datasets remain limited, especially outside common tasks, while controlled collections and simulated missing modalities inadequately reflect real-world conditions.
  • Evaluation and diagnostics: Evaluation should go beyond unimodal baselines and standard metrics by measuring each modality’s contribution, noise tolerance, pairing requirements, domain shifts, interpretability, and fairness.
  • Alignment: Manual alignment of audio, video, and text signals requires substantial preprocessing effort and restricts real-time use.
  • Noisy modalities: Current models lack uniform robustness practices for varied noisy conditions, while noisy labels continue to affect model robustness and performance.
  • Future directions: The survey highlights missing-modality support during training and testing, mechanisms to measure co-learning, and fairness and bias as key open problems.

8. Conclusion

The paper concludes with a systematic survey and detailed taxonomy of multimodal co-learning objectives, implementations, applications, datasets, open problems, and future directions. The taxonomy is intended to help researchers assess coverage of co-learning aspects and guide future work.

  • The survey systematically reviews multimodal co-learning objectives, taxonomy, implementation methods, applications, datasets, open problems, and future directions.
  • Its taxonomy covers modality presence, data parallelism, noisy modalities, modality annotations, domain adaptation, and interpretability and fairness.
  • Representative studies are identified by their implemented objectives and involved modalities, while recent multimodal methods are highlighted as future directions.
Loading 2107.13782v3…