Balancing Multimodal Learning via Functional Progress

1 Introduction

• Modality imbalance arises when modalities learn at different rates, allowing an easier-to-learn modality to dominate while others remain under-optimized.Adding modalities does not always improve performance.Source: §1 · p. 1
• Prediction- and optimization-derived scores may misread intrinsic cross-modal uncertainty and learning-dynamic differences as progress gaps, misguiding rebalancing.Such disparities may not indicate which modality is under-optimized or how strongly it should be adjusted.Source: §1 · p. 1
• FGMO estimates Relative Functional Progress (RFP) by calibrating each modality’s update-induced function-space response against its loss-aligned unimodal reference.This makes progress signals more comparable across modalities while accounting for modality-specific learning dynamics.Source: §1 · pp. 1–2
• RFP-guided interventions remain beneficial at most checkpoints, while RGM- and SMV-guided interventions become harmful later despite persistent audio–video score gaps.In controlled CREMA-D continuations, RFP’s gap narrows and eventually reverses its ordering.Source: §1 · p. 2, Figure 1 · p. 2
• Functional Progress Estimation and Functional Response Control form a closed loop that allocates modality-level response budgets and adjusts tensor-wise learning rates.The paper also reports a one-step target-contraction property under bounded controller-state mismatch and effectiveness across six multimodal benchmarks.Source: §1 · pp. 2–3

Imbalanced Multimodal Learning

• Prior rebalancing methods use prediction-related signals or optimization-related statistics, but their modality-wise scores are not inherently comparable across modalities.The section describes prediction-based approaches including OGM, PMR, AGM, ARL, and AUG, and optimization-based approaches including G-Blend, Greedy, AMSS+, and RGM.Source: §2 · p. 3

Function-Space Methods

• Function-space research has examined optimization distances, output regularization, sequential-learning parameterizations, and estimators of update-induced function changes, mainly for single-network or unimodal learning.How to use function space to measure and balance modality competition in joint multimodal training remains under-explored.Source: §2 · p. 3

3.1 Multimodal Learning

• The setup uses modality-specific encoders and classifiers, averages modality logits for joint prediction, and optimizes joint plus unimodal losses.Source: §3.1 · p. 4

3.2 Functional Progress Estimation

• FPE recovers unit-learning-rate updates, estimates tensor-wise function-space responses, and compares joint-training responses with loss-matched unimodal references to obtain RFP.Source: §3.2 · p. 5

3.3 Functional Response Control

• FRC uses log RFP and a KL constraint on prior allocation to redistribute response toward slower-progress modalities, then maps targets to tensor-wise learning rates.Source: §3.3 · p. 6

3.4 Theoretical Analysis

• With 0 < λ < 2 and bounded controller-state mismatch, FRC’s assigned target ratio strictly contracts toward balanced relative progress.Source: §3.4 · pp. 6–7

4.1 Experimental Setup

• Experiments cover six audio-video, image-text, and trimodal datasets, comparing fusion and rebalancing baselines using accuracy, mAP, and Macro-F1.Source: §4.1 · p. 7

4.2 Main Results

• FGMO achieves the best results on both reported metrics across all six datasets; on KSounds, it reaches 0.7565 accuracy and 0.8210 mAP, exceeding RGM by 2.56% and 1.22%, respectively.Simple fusion underperforms the best unimodal model on Twitter, while rebalancing baselines vary across datasets and metrics.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 Ablation Study

• Removing function-space measurement lowers accuracy by 1.92% on Sarcasm and 7.75% on KSounds; replacing RFP lowers it by 1.99% and 6.13%, while epoch alignment and removing tensor-wise allocation also hurt.Combining all four designs achieves the best multimodal and modality-specific performance on both datasets.Source: §4.3 · pp. 8–9, Table 2 · p. 9

4.4 Hyperparameter Analysis

• K = 20 achieves 0.8589 accuracy and a 2.1796-hour training time on CREMA-D; smaller intervals increase probing overhead, while larger ones delay correction.The authors use K = 20 as the default, balancing feedback stability, responsiveness, and training efficiency.Source: Figure 3 · p. 9, §4.4 · p. 9
• Within 0 < λ < 2, accuracy remains stably high and RFP discrepancies are smaller; λ ≥ 2 degrades performance and larger gains accompany pronounced oscillations.Accuracy peaks at λ = 1, with reported values of 0.8589 and 0.8547.Source: Figure 4 · p. 9, §4.4 · p. 10

4.5 Further Analysis

• Across six datasets and 1,000 checkpoints, RFP has the highest beneficial rate, 83.7%, for interventions that improve accuracy over vanilla continuations.The comparison evaluates guidance reliability across training stages against five baselines.Source: Figure 5 · p. 10, §4.5 · p. 10
• FRC keeps stem-layer learning-rate multipliers near 1×, suppresses intermediate and deep blocks, then raises them at prediction heads, with distinct deeper-block patterns across audio and video.Source: Figure 5 · p. 10, §4.5 · p. 10
• Reference construction increases end-to-end training cost, although fixed references can be reused across seeds and the reported joint-training comparison excludes construction.FGMO also adds periodic function-space probing while improving multimodal performance through reference-calibrated rebalancing.Source: §4.5 · p. 10

• FGMO alleviates modality imbalance through comparable progress estimation and coordinated optimization.Source: §5 · p. 10
• FPE calibrates update-induced function-space responses against loss-aligned unimodal references to produce Relative Functional Progress (RFP).Source: §5 · p. 11
• FRC redistributes modality-level function-space budgets and realizes target responses through tensor-wise learning-rate adjustment.Theoretical analysis establishes a one-step target-contraction property under bounded controller-state mismatch; experiments demonstrate FGMO's effectiveness across multiple multimodal benchmarks.Source: §5 · p. 11

• GPT assisted only with language polishing and readability; the authors reviewed and verified all edits and take responsibility for the final content.Source: AI Use Statement · p. 11