Source-linked AI summary
Balanced Multimodal Learning via On-the-fly Gradient Modulation
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, Di Hu
TL;DR
Multimodal models can under-optimize one modality when another dominates training. OGM-GE dynamically modulates modality gradients and consistently improves performance across four datasets, while remaining a flexible plug-in for existing methods.
Problem
Uniform objectives can leave multimodal representations under-optimized when a better-performing modality dominates optimization, limiting exploitation of multiple modalities.
Method
OGM-GE dynamically monitors modality contribution discrepancies and modulates gradients to allocate more optimization to under-optimized modalities, while enhancing generalization.
Results
OGM-GE consistently improves performance across four multimodal datasets and works as a flexible plug-in for vanilla fusion methods, specialized frameworks, and existing multimodal models.
Takeaways & Limitations
The strategy offers a versatile way to improve multimodal methods without requiring additional uni-modal classifiers for gradient modulation.
Takeaways & Limitations
The supplied limitation passage is an incomplete technical footnote and does not specify a substantive limitation or scope boundary.
Abstract
from arXiv · showhide
Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, vision in drawing picture event, etc. To alleviate this optimization imbalance, we propose on-the-fly gradient modulation to adaptively control the optimization of each modality, via monitoring the discrepancy of their contribution towards the learning objective. Further, an extra Gaussian noise that changes dynamically is introduced to avoid possible generalization drop caused by gradient modulation. As a result, we achieve considerable improvement over common fusion methods on different multimodal tasks, and this simple strategy can also boost existing multimodal methods, which illustrates its efficacy and versatility. The source code is available at \url{https://github.com/GeWu-Lab/OGM-GE_CVPR2022}.
1. Introduction
The paper identifies optimization imbalance in multimodal models: a dominant modality can leave other modality representations under-optimized, limiting the exploitation of multimodal information. It proposes OGM-GE, which dynamically modulates modality optimization and enhances generalization, improving vanilla and existing multimodal frameworks.
- Optimization imbalance: Multimodal models optimizing a uniform objective can underperform unimodal models, despite having access to more views.This challenges the expectation that integrating modalities should at least match unimodal performance.
- Optimization imbalance: Even when joint multimodal performance is best, its individual modality representations can be worse than those from corresponding unimodal models.On VGGSound, the joint model has the best event classification performance, while both its visual and audio modalities perform worse individually.
- Optimization imbalance: Dataset preferences can make one modality dominant, causing a larger accuracy drop in the other modality and creating optimization imbalance.VGGSound favors audio, and the visual modality drops more markedly than the audio modality.
- Method: The dominant modality contributes lower joint discriminative loss and uses limited propagated gradients to control optimization, leaving the other modality under-optimized.The paper analyzes the imbalance from an optimization perspective and motivates On-the-fly Gradient Modulation.
- Method: OGM-GE dynamically controls each modality’s optimization while enhancing generalization to address optimization imbalance.The method is designed to improve under-optimized representations and avoid generalization degradation.
- Results: OGM-GE can be integrated with vanilla fusion strategies and existing multimodal frameworks, bringing consistent improvement across them.The authors present this as evidence of the method’s versatility.
2. Related works
Related work spans multimodal representation learning, analyses of modality discrepancies, and the role of stochastic gradient noise in optimization and generalization. Prior studies link limited multimodal gains to differing modality convergence rates and investigate Gaussian noise as a complementary optimization mechanism.
- Multimodal learning: Multimodal learning research includes unsupervised correspondence modeling to learn meaningful representations for downstream tasks.The field has attracted increasing attention because multimodal data contains abundant correlation.
- Modality discrepancy: Studies report that multimodal methods may fail to improve performance because modalities differ in their learning behavior.Wang et al. found that different modalities have different convergence rates, hindering jointly trained models from matching or outperforming unimodal counterparts.
- Gradient noise: SGD gradient noise is associated with deep-model generalization and can act as regularization that helps escape saddle points or local optima.The noise arises from random mini-batch sampling, while theoretical work studies convergence with proper Gaussian noise.
3. Method
The method identifies optimization imbalance in multimodal discriminative models and introduces OGM to modulate each modality’s gradient according to contribution discrepancy. It further adds dynamically changing Gaussian noise through GE to enhance generalization.
- Optimization imbalance: A better-performing modality can dominate optimization, leaving the other modality with under-optimized representations near multimodal convergence.The dominant modality contributes more to lowering the global loss, limiting backpropagation updates for the weaker modality’s parameters.
- On-the-fly Gradient Modulation: The proposed OGM strategy monitors each modality’s contribution discrepancy and adaptively modulates its gradient during SGD.The discrepancy ratio ρ^u_t estimates relative uni-modal contribution, while coefficient k^u_t controls the update.
- On-the-fly Gradient Modulation: When ρ^u_t > 1, OGM mitigates optimization of the better-performing modality while leaving the other modality unaffected.This gives the weaker modality adequate training and alleviates the optimization imbalance.
- Generalization Enhancement: GE adds randomly sampled Gaussian noise to enhance SGD noise and address possible generalization degradation caused by gradient modulation.The paper relates larger SGD noise and higher gradient covariance to better generalization ability.
4. Experiments
Experiments across audio-visual datasets show that OGM-GE improves conventional fusion methods, competing modulation strategies, and representative localization frameworks. Analyses further indicate more balanced modality optimization and compatibility with both SGD and Adam.
- Combination with conventional fusion methods: OGM-GE consistently improves vanilla fusion methods, including baseline, concatenation, and summation, as well as FiLM across evaluated datasets.The experiments evaluate OGM-GE with conventional and specifically designed fusion methods on CREMA-D and VGGSound.
- Combination with conventional fusion methods: Audio-only performance exceeds every vanilla fusion method on CREMA-D, indicating that multimodal fusion can suppress a stronger modality.After applying OGM-GE, all vanilla fusion methods gain considerable performance across different datasets.
- Comparison with other modulation strategies: OGM-GE achieves the best performance among the compared modulation methods, and adding GE produces the best results on all evaluated datasets.The comparison uses the same ResNet18-based backbone with concatenation and includes Modality-Dropout and Gradient-Blending.
- Audio-visual event localization: OGM-GE improves two representative AVEL frameworks, AGVA and PSP, on the AVE dataset, showing consistent gains despite cross-modal interactions.The result supports the claim that modality imbalance occurs across different audiovisual architectures.
- Imbalance modulation analysis: After modulation, both audio and visual modalities improve, while OGM-GE starts behind competing methods but surpasses them by the end of training.The analysis attributes this behavior to reducing optimization of the better-performing modality through gradient modulation; Figure 3 describes more balanced optimization.
- Adaptation to other optimizer: OGM-GE improves models optimized with both SGD and Adam, while SGD performs better than Adam under the same experimental setting.The optimizer study applies OGM-GE to Adam and compares the resulting models with corresponding SGD models.
5. Discussion
The paper proposes OGM-GE, a simple multimodal learning strategy that addresses optimization imbalance and facilitates exploitation of both modalities. It delivers consistent gains across four representative datasets and serves as a flexible plug-in for multiple fusion approaches.
- Method and contribution: OGM-GE is proposed as a simple, effective strategy for multimodal learning.Its stated purpose is to address the optimization imbalance problem.
- Method and contribution: The strategy alleviates optimization imbalance, facilitating exploitation of both modalities.
- Empirical scope: OGM-GE achieves consistent performance gains on four representative multimodal datasets under various settings.
- Applicability: The method generally functions as a flexible plug-in for vanilla and specifically designed fusion methods.