CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery

Mahir Shahriar Tamim; Sharjil Khan; Md. Samiul Alim; Tanvir Ahmed Khan; Shafin Rahman; Nabeel Mohammed

Source-linked AI summary

CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery

Mahir Shahriar Tamim, Sharjil Khan, Md. Samiul Alim, Tanvir Ahmed Khan, Shafin Rahman, Nabeel Mohammed

arXiv:2608.24947v1Aug 24, 2026cs.LGcs.AI

TL;DR

Multimodal training can suffer from modality imbalance, unstable gating, and conflicting fusion gradients. CAT-GS controls these dynamics during backpropagation, improving or matching fused accuracy and producing smoother gating across diverse benchmarks.

Problem
Multimodal training lacks a unified way to address modality imbalance, unstable gating, and conflicting gradients at shared fusion layers.
Method
CAT-GS is an optimization-level controller that uses stabilized reliability and regime-aware gating, gradient-budget renormalization, and fusion-only conflict resolution.
Results
Across audio–visual, tri-modal, synthetic, and additional benchmarks, CAT-GS improves or matches fused accuracy while yielding smoother optimization and fewer conflicting fusion gradients.
Takeaways & Limitations
CAT-GS provides a broadly applicable approach for more balanced and stable multimodal learning without modifying architectures or task losses.
Takeaways & Limitations
CAT-GS offers limited benefit on large-scale VGGSound, where weak teacher reliability and large label spaces favor representation capacity and data scale over gradient-level control.

Abstract

from arXiv · show

End-to-end training of multimodal neural networks often exhibits unstable neural dynamics characterized by three coupled failure modes that degrade learning: (i) modality imbalance, where one branch dominates gradient-based optimization; (ii) unstable gating, where noisy confidence cues induce erratic modality selection; and (iii) fusion interference, where modality-specific gradients conflict at the shared fusion layer. We propose CAT-GS (Calibrated, Adaptive, Thresholded Gating with Fusion Surgery), a neural dynamics-based optimization controller for intelligent computing applications. CAT-GS operates during backpropagation without modifying model architectures, fusion modules, or task losses. Through calibration of teacher-derived reliability via temperature scaling and EMA smoothing, CAT-GS stabilizes neural dynamics using a margin-thresholded policy to switch between warm-up dropout, weak-modality prioritization, and weak-biased blending, stabilizes gradient magnitudes under aggressive gating via capped gradient-budget renormalization, and applies fusion-only PCGrad to reduce destructive cross-modal interference at the primary shared bottleneck. We evaluate CAT-GS on audio--visual multimodal pattern recognition benchmarks (CREMA-D, AV-MNIST, and VGGSound), a tri-modal setting (UR-FUNNY), controlled synthetic data (CG-MNIST), and additional cross-domain benchmarks (AVE and CMU-MOSI). CAT-GS improves or matches fused multimodal accuracy against strong imbalance-aware baselines (including OGM-GE, G$^2$D, and UMT) across settings, and yields smoother gating behavior with fewer conflicting fusion gradients.

1. Introduction

CAT-GS is a neural-dynamics optimization controller for multimodal learning that regulates backward-stage signals to address coupled imbalance, gating instability, and fusion interference. It combines calibrated regime-based gating, budget-preserving gradient renormalization, and fusion-only gradient surgery, with validation across diverse multimodal benchmarks.

Core framework: CAT-GS regulates multimodal learning dynamics during backpropagation without modifying model architectures or task losses.It wraps an existing multimodal training loop and modifies only backward-stage optimization signals.
Adaptive gating: Calibrated reliability margins drive interpretable transitions among warm-up dropout, dominance suppression, and weak-biased blending, avoiding oscillatory gating.The controller uses knowledge-distillation reliability and margin thresholds to select gating regimes.
Gradient stabilization: Budget-preserving gradient renormalization stabilizes update magnitudes under aggressive hard gating using an EMA target magnitude and hard cap.This mechanism mitigates prolonged imbalance and brittle convergence.
Fusion surgery: Fusion-only PCGrad reduces destructive cross-modal interference at the shared fusion bottleneck while preserving unimodal encoder dynamics at lower cost than global projection.The surgery is applied exclusively where cross-modal gradients first interact.
Empirical validation: CAT-GS improves fused accuracy and optimization stability across audio–visual, tri-modal, controlled synthetic, AVE, and CMU-MOSI benchmarks.Reported behavioral evidence includes smoother gating dynamics and fewer negative fusion-gradient cosine events.

2. Related Work

Prior work addresses modality imbalance, confidence-based fusion, and gradient conflicts through largely separate techniques, while CAT-GS integrates calibrated gating, budget-preserving rescaling, and fusion-localized projection into one optimization-stage controller. It also addresses limitations of coarse dominance cues and uncalibrated teacher signals through confidence margins, temperature scaling, and EMA stabilization.

Modality Imbalance in Multimodal Learning: Earlier imbalance methods use auxiliary losses, feature-norm regularization, gradient rescaling, or multi-loss balancing to prevent one modality from dominating updates.Later methods include MMCosine, DRL, and MCR, but may rely on coarse dominance cues or leave suppressed branches vulnerable to long-horizon gradient starvation.
Confidence-Based Gating and Dynamic Fusion: Multimodal fusion research spans sentiment analysis, video captioning, and adaptive gating, while teacher-guided multimodal training commonly uses unimodal teacher outputs for supervision.UMT matches logits and features but treats teacher outputs as infallible, overlooking miscalibration.
Confidence-Based Gating and Dynamic Fusion: CAT-GS uses calibrated confidence margins to make suppression temporary and pairs modulation with budget-preserving rescaling.This contrasts with approaches that use predefined or coarse dominance cues without explicitly preventing gradient starvation in suppressed branches.
Confidence-Based Gating and Dynamic Fusion: CAT-GS embeds temperature scaling and EMA stabilization within distillation-guided gating to produce reliable signals for adaptive optimization.The approach addresses the limitation of treating teacher outputs as infallible and distinguishes online integration from post-hoc calibration.
Integrated Optimization Control: CAT-GS integrates imbalance handling, confidence-based fusion, and gradient-conflict mitigation into one optimization-stage controller across fusion designs.Its contribution includes fusion-localized projection and budget-preserving renormalization.

3. Methodology

CAT-GS is an optimization-level controller for multimodal training that operates on gradients without changing the model architecture or forward computation. It stabilizes modality reliability, gating, update magnitudes, and fusion gradients through calibrated control and fusion-only surgery.

Training iteration: The training iteration applies regime-based gating, gradient-budget stabilization, and fusion-only surgery before the optimizer update.This sequence is designed to preserve weaker-modality learning while reducing interference at the shared fusion bottleneck.
Optimization-level controller: CAT-GS operates on gradients produced by a generic multimodal model with trainable modality encoders, a fusion network, and frozen unimodal teachers.The controller is objective-agnostic and can be applied to multimodal models with modality-specific parameters.
Controller components: Four coupled components stabilize training: calibrated reliability estimation, margin-thresholded adaptive gating, gradient-budget stabilization, and fusion-layer gradient surgery.Together, they address unstable confidence signals, modality dominance, aggressive-gating magnitude changes, and destructive cross-modal conflicts.
Reliability calibration: Teacher reliability uses batch-mean confidence signals and EMA smoothing to reduce batch-to-batch volatility and gate thrashing.Temperature scaling is optional; the main experiments use Tm=1 unless explicitly stated, while high momentum such as β = 0.9 enables gradual reactions.
Adaptive gating: Smoothed reliabilities define a margin between modalities, and thresholds τlow and τhigh select among three regimes whose coefficients αm scale modality gradients.The regime map adapts gradient routing to differences in calibrated confidence.

1. Warm-up Dropout (∆< τlow, early epochs)

During early epochs with Δ < τlow, CAT-GS applies stochastic modality dropout because teacher signals are noisy. This short warm-up prevents premature specialization and strengthens unimodal representations.

1. Warm-up Dropout (∆< τlow, early epochs): Noisy early teacher signals motivate stochastic modality dropout during the warm-up phase.The dropout probability is denoted pdrop.
1. Warm-up Dropout (∆< τlow, early epochs): CAT-GS uses dropout for a short warm-up to prevent premature modality specialization.
1. Warm-up Dropout (∆< τlow, early epochs): The warm-up dropout strengthens unimodal representations before later gating decisions.

2. Dominance Suppression (∆> τhigh)

When one modality is clearly more reliable, CAT-GS uses hard gating to protect the weaker branch and counteract runaway dominance. This regime applies only when the reliability margin exceeds the high threshold; otherwise, CAT-GS uses soft blending.

Dominance Suppression (∆> τhigh): CAT-GS performs hard gating when one modality becomes clearly more reliable, protecting the weaker branch.The weaker modality receives full update emphasis during this regime.
Dominance Suppression (∆> τhigh): The controller applies hard gating only when the reliability margin is clearly large; otherwise, it uses soft blending.

3. Weak-Bias Blending (τlow ≤∆≤τhigh)

When modality reliabilities are close, CAT-GS uses smooth, weak-biased blending with a nonzero floor so both branches remain active while favoring the weaker modality. This policy is paired with gradient-budget stabilization to prevent aggressive gating from collapsing optimization updates.

Weak-Bias Blending: CAT-GS uses smooth weak-biased blending when modality reliabilities are close, avoiding abrupt switching.The weak modality receives a controlled advantage through bias-adjusted blending weights.
Weak-Bias Blending: Bias-adjusted weights increase the weaker modality by 1 + λbias and decrease the stronger modality by 1 −λbias before normalization.The weights are normalized with floor ε so both modalities remain active.
Gradient Stabilization: A nonzero floor ε prevents consistently down-weighted modalities from suffering gradient starvation and becoming permanently disabled.Without such protection, near-zero gradients can stagnate the suppressed modality’s features and further reduce its future reliability.
Gradient Stabilization: Gradient-budget reallocation stabilizes the active branch during hard gating by renormalizing encoder gradients toward their EMA magnitude, capped by γcap.When αm = 0, the gated modality receives no update by design, while γcap limits excessive rescaling.

4. Experiments

CAT-GS is evaluated across audio–visual, tri-modal, synthetic, and cross-domain benchmarks, generally matching or improving strong imbalance-aware baselines on fused accuracy. Its benefits are accompanied by smoother gating, reduced gradient interference, and limited gains in noisy large-scale settings such as VGGSound.

Audio–Visual Classification Benchmarks: CAT-GS improves CREMA-D fused accuracy to 86.29%±0.15 versus 85.89% for G2D and achieves the best AV-MNIST fused accuracy at 73.21%±0.08.The controller remains competitive when modality imbalance is mild and improves performance when video is substantially stronger than audio.
Audio–Visual Classification Benchmarks: On VGGSound, CAT-GS reaches 53.24%±0.24 fused accuracy, below G2D at 53.82% and UMT at 53.78%.The large label space, noisy clips, and weakly informative teacher reliabilities limit the benefit of gradient-level control.
Tri-Modal Humor Understanding (UR-FUNNY): In UR-FUNNY, CAT-GS is competitive in audio–visual settings and consistently achieves the best multimodal accuracy after textual modality is introduced.The text-inclusive setting creates richer modality interactions than the bi-modal configuration.
Additional Benchmarks (AVE and CMU-MOSI): CAT-GS remains strongest on AVE and CMU-MOSI, reaching 78.3%±1.0 on MOSI three-modality classification versus 77.99 for DRL and 76.5 for MCR.Reported 95% confidence intervals are [73.70, 74.70] on AVE, [73.02, 78.98] on CMU-MOSI (V–T), and [75.82, 80.78] on CMU-MOSI (V–A–T).
Controlled Synthetic Benchmark (CG-MNIST): CAT-GS maintains strong fused performance on CG-MNIST despite early spurious predictive strength in one modality and avoids brittle training dynamics observed in several baselines.The controlled results support margin-thresholded gating and budget-aware stabilization.
Optimization Dynamics: CAT-GS produces the smoothest gating trajectories, maintains positive average fusion-layer cosine similarity, and substantially reduces batches with negative cosine similarity.EMA-smoothed reliability, margin-based switching, and stabilization mechanisms suppress gate thrashing and destructive cross-modal interference.

5. Conclusion

CAT-GS is an optimization-stage controller for balanced and robust multimodal learning that treats training as a regime-switching control problem. It stabilizes gradient flow, update magnitude, and fusion-gradient geometry without modifying architectures or designing bespoke losses.

CAT-GS is an optimization-stage learning-dynamics controller for balanced and robust multimodal learning.
The method treats multimodal training as a regime-switching control problem rather than modifying model architectures or designing bespoke losses.
CAT-GS enforces stability constraints on gradient flow, update magnitude, and fusion-gradient geometry across multiple multimodal benchmarks.The passage names CREMA-D, AV-MNIST, AVE, CMU-MOSI, and UR-FUNNY among the evaluated benchmarks.