Source-linked AI summary
PMR: Prototypical Modal Rebalance for Multimodal Learning
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, Song Guo
TL;DR
Multimodal learning can suffer from modality imbalance because a dominant modality disrupts slower modalities under unified optimization. PMR uses prototypes with prototypical cross-entropy to accelerate the slow modality and prototypical entropy regularization to reduce early suppression. The method achieves considerable improvements across three multimodal datasets while remaining independent of model structure and fusion method.
Problem
Modality imbalance leaves individual modalities below their potential because unequal learning rates let dominant modalities suppress slower modalities.
Method
PMR uses class prototypes and non-parametric classifiers to evaluate modalities, accelerates slow modalities with PCE, and applies early-stage PER to alleviate dominant-modality suppression.
Results
PMR achieves considerable performance improvement on three multimodal datasets with different model structures and fusion methods.
Takeaways & Limitations
Because PMR relies on modality representations rather than particular structures or fusion methods, its prototype-based classifiers can be applied across multimodal scenarios.
Takeaways & Limitations
The logits-based performance estimation requires classifier outputs decomposable into separate modality components, limiting its application in many scenarios.
Abstract
from arXiv · showhide
Multimodal learning (MML) aims to jointly exploit the common priors of different modalities to compensate for their inherent limitations. However, existing MML methods often optimize a uniform objective for different modalities, leading to the notorious "modality imbalance" problem and counterproductive MML performance. To address the problem, some existing methods modulate the learning pace based on the fused modality, which is dominated by the better modality and eventually results in a limited improvement on the worse modal. To better exploit the features of multimodal, we propose Prototypical Modality Rebalance (PMR) to perform stimulation on the particular slow-learning modality without interference from other modalities. Specifically, we introduce the prototypes that represent general features for each class, to build the non-parametric classifiers for uni-modal performance evaluation. Then, we try to accelerate the slow-learning modality by enhancing its clustering toward prototypes. Furthermore, to alleviate the suppression from the dominant modality, we introduce a prototype-based entropy regularization term during the early training stage to prevent premature convergence. Besides, our method only relies on the representations of each modality and without restrictions from model structures and fusion methods, making it with great application potential for various scenarios.
1. Introduction
Multimodal learning can outperform single-modal learning, but modality imbalance leaves individual modalities below their potential because a dominant modality disrupts slower modalities. PMR uses prototypes to accelerate the slow modality independently and reduce dominant-modality suppression.
- Problem: Modality imbalance arises because modalities overfit and converge at different rates while a dominant modality hinders full multimodal feature utilization.Existing methods often optimize different modalities with a unified objective despite their inconsistent learning efficiency.
- Problem: The dominant modality suppresses slower modalities by interfering with both their learning rates and gradient update directions.This interference makes it difficult for slower modalities to improve their unimodal performance.
- PMR: PMR introduces class prototypes as representative embeddings and uses them to build non-parametric classifiers for modality evaluation and feature clustering.The method also derives a prototype-based metric to monitor modality imbalance during training.
- PMR: PMR accelerates the slow-learning modality with prototypical cross-entropy while slowing the dominant modality through early-stage prototypical entropy regularization.These components target feature exploitation and suppression alleviation separately.
- Scope and results: PMR relies only on modality representations, avoiding restrictions from model structures and fusion methods.The authors report considerable improvements over existing methods and strong generality across application settings.
2. Related Works
Related work frames modality imbalance as a consequence of unequal overfitting and generalization rates, with dominant modalities controlling joint optimization. Prior remedies add classifiers, distillation, or pathway changes, while prototypes provide a non-parametric alternative for modality evaluation.
- Modal imbalance: Different modalities overfit and generalize at different rates, so unified optimization can produce suboptimal multimodal solutions dominated by the better modality.Prior work also reports that many multimodal networks fail to outperform the best single-modal network.
- Prior approaches: Existing approaches use additional classifiers, knowledge distillation, or pathway dropping to address modality imbalance.These methods target gradient mixing, unimodal improvement, or modality-specific learning behavior.
- Prior approaches: Several prior methods introduce extra model structure and computational effort, increasing training complexity and expense.This is identified as a limitation of classifier- and distillation-based approaches.
- Prototypical networks: Prototypical networks represent classes with embeddings surrounded by same-class points and have been used in few-shot, zero-shot, long-tail, domain-adaptation, and unsupervised settings.Their prototypes can function as non-parametric classifiers.
- Prototypical networks: In this paper, prototypes are used to construct non-parametric classifiers that evaluate each modality’s features.This avoids requiring the parametric linear classifiers described in the related work.
3. Modality Imbalance Analysis
The analysis attributes modality imbalance to unequal learning and gradient influence, where the dominant modality suppresses the slower modality and disrupts its preferred update direction.
- Model formulation: The multimodal model uses two unimodal encoder branches whose representations are fused before a linear classifier produces logits.The branches extract modality-specific features, and the fusion operation connects them to the classifier.
- Gradient analysis: The fused cross-entropy objective produces gradients whose contribution is influenced by the performance of the fused modality.The formulation does not directly reveal each modality’s individual contribution to the update.
- Empirical imbalance: In CREMA-D, audio performance closely tracks multimodal performance while visual performance is much worse, indicating audio dominance and visual suppression.The audio modality’s greater similarity to the fused performance corresponds to excessive dominance in gradient updates.
- Empirical imbalance: Increasing the slow modality’s gradient magnitude improves validation accuracy only slightly because other modalities still disturb its feature exploitation.The gradient direction from multimodal output increasingly diverges from each modality’s guidance direction during training, while remaining acute.
4. Prototypical Modal Rebalance
PMR evaluates each modality with prototype-based classifiers, accelerates the slower modality through prototypical cross-entropy, and reduces dominant-modality inhibition with early entropy regularization.
- Prototype-based evaluation: PMR introduces class prototypes as representative embeddings and uses distances to them to build non-parametric unimodal classifiers.Prototype-based class distributions support modality performance estimation and imbalance monitoring without relying on decomposable classifier logits.
- Prototype-based evaluation: The prototype-based imbalance metric uses batch representations and prototypes to evaluate modality imbalance in real time independently of fusion and classifier structure.The distance function is Euclidean in this formulation.
- Modal acceleration: PCE accelerates the slow-learning modality by enhancing its clustering toward prototypes, while the acceleration loss combines CE and PCE terms.The modulation strength is controlled by α, and coefficients β and γ are adjusted using the dynamic imbalance ratio ρt.
- Modal acceleration: The slower modality receives facilitation while the better modality retains its original learning strategy, and PCE depends only on modality representations.This representation-level design supports application across fusion scenarios with modality encoders.
- Inhibition reduction: PER slows the dominant modality’s convergence during the first few epochs to reduce early inhibition and prevent premature convergence.The entropy regularization uses softmax probabilities and entropy, with β and γ applied to the opposite modality relative to the acceleration loss.
- Training procedure: The PMR algorithm computes subset representations and prototypes, estimates ρt for each minibatch, and selects the final loss with or without early regularization.Prototype computation and loss updates are repeated within the training procedure.
5. Evaluation
Across audio-visual and synthetic multimodal benchmarks, PMR improves multimodal performance across fusion methods, architectures, tasks, and optimizers while addressing modality imbalance. Ablations show that PMR particularly benefits the slow-learning modality, remains effective after dominant-modality convergence, and is stable when prototype subsets exceed 10%.
- Conventional fusion methods: PMR gets great performance improvement on nearly all scenarios across CREMA-D, AVE, and Colored-and-gray MNIST with varied fusion methods.Table 1 covers concatenation, summation, FiLM, and gated fusion, with unimodal baselines included.
- Comparison with modulation strategies: PMR achieves the best performance among Modality-Drop, Gradient-Blending, and OGM-GE with concatenation and FiLM fusion.The authors attribute the main improvement to PCE, while PER adds almost 1% or maintains performance.
- Different architectures: PMR improves performance with MMTM and CentralNet on CREMA-D and Colored-and-gray MNIST, supporting applicability across intermediate fusion architectures.These experiments evaluate the methods with and without PMR.
- Another task: PMR is also evaluated for audiovisual event localization on AVEL using concatenation and DMRN fusion methods.The experiment applies PMR to the ground-truth modality and evaluates supervised event localization in a late-fusion style.
- Uni-modal performance comparison: On CREMA-D, PMR considerably improves the visual branch while leaving the audio branch largely unchanged, and gradually decreases the imbalance ratio ρ.The imbalance ratio continues decreasing even after the dominant modality has converged, indicating that PCE is less affected by other modalities.
- Prototype subset scale: When prototype subsets exceed 10% of the training data, PMR accuracy tends to stabilize on CREMA-D and AVE, whereas very small subsets can bias prototypes and reduce performance.Prototype computation is the main additional cost, but the required computation can be reduced through reasonable subset selection.
- Adaptive optimizers: PMR works with AdaGrad and Adam, although optimizer performance varies across datasets.Adam performs best on CREMA-D, while SGD performs best on AVE.
6. Discussion
PMR addresses modality imbalance by assigning different learning strategies to slow and dominant modalities. It improves multimodal performance across three datasets while remaining applicable across model structures and fusion methods.
- PMR accelerates the slow modality with prototypical cross entropy and reduces dominant-modality inhibition with prototypical entropy regularization.
- PMR achieves considerable performance improvement on three multimodal datasets using different model structures and fusion methods.
- Prototype-based non-parametric classifiers can be used whenever instance representations are available for each modality.