Source-linked AI summary
What Makes Training Multi-Modal Classification Networks Hard?
Weiyao Wang, Du Tran, Matt Feiszli
TL;DR
The paper investigates why jointly trained multimodal networks can underperform their best unimodal counterparts, attributing the gap to overfitting and mismatched modality generalization. It proposes Gradient Blending, which blends supervision signals according to overfitting behavior, and reports gains over baselines plus state-of-the-art accuracy across multiple benchmarks.
Problem
Jointly trained multimodal networks often underperform the best unimodal network because increased capacity and differing modality overfitting rates make a single optimization strategy sub-optimal.
Method
Gradient Blending minimizes the overfitting-to-generalization ratio by optimally blending multiple modality supervision signals during training.
Results
Gradient Blending outperforms unimodal and naive joint-training baselines and achieves state-of-the-art accuracy on Kinetics, EPIC-Kitchen, and AudioSet.
Takeaways & Limitations
Multimodal end-to-end training benefits from principled measurement and correction of overfitting rather than relying only on standard regularizers or fusion architectures.
Takeaways & Limitations
The blending analysis assumes cross terms are small relative to the expected component terms; when overfitting is highly correlated, blending may offer little gain.
Abstract
from arXiv · showhide
Consider end-to-end training of a multi-modal vs. a single-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its single-modal counterpart. In our experiments, however, we observe the opposite: the best single-modal network always outperforms the multi-modal network. This observation is consistent across different combinations of modalities and on different tasks and benchmarks. This paper identifies two main causes for this performance drop: first, multi-modal networks are often prone to overfitting due to increased capacity. Second, different modalities overfit and generalize at different rates, so training them jointly with a single optimization strategy is sub-optimal. We address these two problems with a technique we call Gradient Blending, which computes an optimal blend of modalities based on their overfitting behavior. We demonstrate that Gradient Blending outperforms widely-used baselines for avoiding overfitting and achieves state-of-the-art accuracy on various tasks including human action recognition, ego-centric action recognition, and acoustic event detection.
1. Introduction
End-to-end multimodal training can underperform the best unimodal model despite access to more information. The paper attributes this gap to overfitting and evaluates regularization and fusion strategies before introducing Gradient Blending.
- Problem: The best unimodal network often outperforms the jointly trained multimodal network across modalities and datasets.This contradicts the expectation that a well-optimized multimodal model should match or exceed unimodal solutions.
- Problem: Multimodal networks show higher training accuracy but lower validation accuracy, consistent with overfitting from increased capacity.The late-fusion audio-visual network has nearly twice as many parameters as the visual network.
- Approach: The paper tests dropout, pre-training, early stopping, mid-level concatenation, and gated fusion as ways to address overfitting or architectural deficiencies.The fusion variants include Squeeze-and-Excitation and Non-Local gates.
- Findings: Pre-training and early stopping fail to improve effectively, while mid-concatenation and dropout provide only modest gains over the RGB model.On Kinetics, dropout and mid-concatenation improve over late concatenation by 1.5% and 1.4%, respectively.
- Contribution: Gradient Blending minimizes overfitting through an optimal blend of multiple supervision signals and achieves state-of-the-art accuracy across several multimodal benchmarks.The method is described as task-agnostic and architecture-agnostic, with applications beyond audio-visual inputs.
2. Multi-modal training via Gradient-Blending
The paper formulates uni-modal and late-fusion multi-modal training, then introduces OGR-based Gradient-Blending to combine modality-specific supervision signals while limiting overfitting.
- 2.1. Background: A uni-modal network minimizes classification loss using one modality’s encoder and classifier, while a multi-modal network fuses features from multiple modality-specific encoders.Late fusion concatenates modality features before passing them to a classifier.
- 2.1. Background: The multi-modal formulation contains every uni-modal solution in principle, but practical optimization does not reliably find those solutions.The classifier could mute all modalities except one, yet the paper states that this solution is not found in practice.
- 2.2. Generalizing vs. Overfitting: The overfitting-to-generalization ratio measures the change in overfitting relative to generalization between two training checkpoints.The paper interprets OGR as the ratio of non-generalizable to generalizable information for cross-entropy loss.
- 2.3. Blending of Multiple Supervision Signals by OGR Minimization: Gradient-Blending estimates gradients from individual modality losses and fused features, then blends them to minimize an infinitesimal OGR during optimization.The method attaches classifiers to modality-specific and fused features and combines their gradient estimates.
- 2.4. Use of OGR and Gradient-Blending in practice: Loss reweighting implements Gradient-Blending by producing a weighted sum of modality gradients and recalibrating learning according to each modality’s overfitting and generalization rates.Weights can be estimated using a held-out validation subset, and offline or online versions can be used during training.
3. Ablation Experiments
Ablations show that naive joint training overfits across modality combinations, while Gradient-Blending improves generalization across optimizers, fusion settings, tasks, and modality relevance.
- Overfitting Problems in Naive Joint Training: Naive audio-RGB training has lower training error but higher validation error than video-only training on Kinetics and mini-Sports.Audio overfits most, video least, and the joint model increases overfitting relative to the video-only model.
- Overfitting Problems in Naive Joint Training: Across all four combinations of audio, RGB, and optical flow, naive joint training performs significantly worse in validation accuracy than the best single stream.Training accuracy is almost always higher for naive joint models, indicating a generalization gap.
- Gradient-Blending: Online G-Blend consistently outperforms naive training at each evaluated super-epoch, providing more generalizable training information.The comparison reuses identical model snapshots and evaluates both methods over the following five epochs.
- Gradient-Blending: G-Blend consistently outperforms visual-only and naive audio-visual baselines across SGD, AdaGrad, and Adam optimizers.The method is also reported to improve across modality combinations, architectures, and tasks.
- Class-Level Analysis: G-Blend improves most on audio-correlated classes, while classes with little audio relevance show smaller or negative gains relative to RGB-only.Examples of audio-relevant improvements include clapping and laughing; unloading truck appears among lower-improvement classes.
4. Comparison with State-of-the-Art
Across Kinetics, AudioSet, and EPIC-Kitchens, Gradient-Blending matches or exceeds strong unimodal and multimodal baselines, often using fewer modalities or models.
- Kinetics: On Kinetics, G-Blend improves 1.3% over the RGB model with the same ip-CSN-152 backbone when both models are trained from scratch.It also exceeds Shift-Attention Network by 1.4% while using fewer modalities and no pre-training.
- Kinetics: On Kinetics, G-Blend is on par with SlowFast while being 2x faster, and fine-tuned G-Blend exceeds SlowFast and SlowFast with Non-Local by 1.5% and 0.6%.With IG-65M visual pre-training, G-Blend reaches 83.3% top-1 and 96.0% top-5 accuracy.
- Cross-Benchmark Comparison: Across ablation benchmarks, G-Blend consistently outperforms regularization baselines and unimodal networks, except for comparable performance to auxiliary loss on mini-AudioSet.The mini-AudioSet exception is attributed to G-Blend learning weights similar to equal weights.
- AudioSet: On AudioSet, G-Blend exceeds Multi-level Attention Network and TAL-Net by 5.8% and 5.5% on mAP, respectively.G-Blend uses 10 clips per video, compared with TAL-Net’s 100, while Multi-level Attention Network uses features pre-trained on YouTube100M.
- Efficiency and Model Size: G-Blend achieves strong results with fewer modalities, fewer backbones, and a single model than several published EPIC-Kitchens systems.The comparisons include systems using optical flow, multiple 3D-CNNs, detection models, or ensembles.
5. Discussion
Gradient-Blending measures and corrects overfitting in multi-modal networks, improving performance over baselines across multiple tasks and benchmarks.
- Gradient-Blending uses a principled overfitting measure to obtain significant improvements over baselines.
- Gradient-Blending either outperforms or matches state-of-the-art methods across multiple tasks and benchmarks.
- The method may extend to end-to-end ensemble training and settings requiring calibration of multiple losses.
A. Estimating Weights on Subsets of Data
The overfitting-to-generalization ratio quantifies training-information quality, and Gradient-Blending minimizes it through locally optimized blends of modality gradients.
- A. Estimating Weights on Subsets of Data: Estimated weights remain stable when using 25%, 50%, or 75% of Kinetics, suggesting lower-cost estimation on small subsets.
- B. Understanding OGR: The overfitting-to-generalization ratio measures the quality of information learned during training.For cross-entropy loss, it is the ratio of non-generalizable bits to generalizable bits.
- B. Understanding OGR: Global OGR optimization is expensive and can favor very underfit models, so the method instead solves an infinitesimal blending problem.
- B. Understanding OGR: The method decomposes training gradients into true-gradient and overfitting components to assess each gradient estimate’s contribution.
- B. Understanding OGR: The final OGR aggregates gradient contributions across training steps and can be minimized by selecting an optimal single-step blend.
C. Proof of Proposition 1
The proof derives Gradient-Blending weights by optimizing the expected objective under a normalization constraint, with a covariance-matrix extension for correlated modality errors.
- C. Proof of Proposition 1: The proof reformulates the optimization under an equivalent weight normalization constraint.
- C. Proof of Proposition 1: The proof begins by computing the expectation of the objective before solving for the constrained optimum.
- C. Proof of Proposition 1: Lagrange multipliers yield the optimal weights after differentiating the objective with respect to each modality weight.
- C. Proof of Proposition 1: The remaining normalizing constant is obtained by applying the constraint and rescaling the weights.
- C. Proof of Proposition 1: When modality overfitting terms are correlated, the derivation extends using a covariance matrix Σ rather than independent terms.
- C. Proof of Proposition 1: Multiple runs show that G-Blend result variance is consistent across modalities on the evaluated datasets.
E. Sub-sampling and Balancing Multi-label Dataset
Balancing multi-label data is difficult because one sampled example can contribute to several correlated classes, so naive per-class subsampling is unsuitable.
- Multi-label classes can be correlated, causing one sampled example to add data volume to multiple classes.
- Naive per-class subsampling is difficult for multi-label datasets because samples affect several class counts simultaneously.
- The paper proposes an algorithm to uniformly subsample and balance AudioSet into mini-AudioSet.
F. Details on Model Architectures
The paper evaluates late and mid-level fusion architectures, including concatenation and gating, while examining how fusion depth and feature dimensions affect performance. These architectural variants remain vulnerable to overfitting, motivating careful comparison across fusion designs.
- Late Fusion By Concatenation: Late fusion concatenates each stream’s output feature vector and can apply dropout after concatenation.
- Fusion-Network Depth: Fusion-network depth has competing effects: fewer layers may not learn feature relations fully, while deeper networks overfit more.
- Mid-Level Fusion: Mid-level fusion matches audio and visual feature dimensions through 1-D, 2-D, or 3-D concatenation strategies.Audio and visual streams differ in frequency-time and spatial-temporal dimensions, requiring pooling, tiling, downsampling, or upsampling.
- Mid-Level Fusion: 3-D concatenation after block 3 works best on Kinetics, whereas 2-D concatenation works best on AudioSet and uses fewer GFLOPs than 3-D concatenation.
- Gating: Gating architectures use one stream to generate channel weights for another, with SE gates providing channel-level modulation and NL gates using Query-Key-Value attention.NL-Gate works best after block 4 when 2-D audio-RGB features provide Key and Value and visual features provide Query.
G. Additional Ablation Results
Additional ablations compare Gradient Blending with an oracle that selects the best modality separately for each class. Gradient Blending remains competitive with this stronger class-wise selection baseline.
- Class-Wise Oracle: 74.1% Top-1 video accuracy from an oracle selecting the best of audio, RGB, and naive A/V per class is 0.6% below offline G-Blend.The oracle selects the modality with the highest validation performance for each class.
G.2. Training Accuracy
Training-accuracy analyses confirm that naive multimodal joint training often fits the training data better while generalizing worse. Early stopping reduces overfitting but shifts the problem toward underfitting.
- Training Accuracy: Across multimodal problems, naive networks have lower validation accuracy than the best single-stream models while almost always achieving higher training accuracy.OF+A is the sole stated exception, with training accuracy similar to the audio network.
- Early Stopping: Using 25%, 50%, or 75% of iterations per epoch reduces overfitting but causes underfitting.
- Early Stopping: The 75% early-stopping schedule performs best among the tested schedules but remains worse than full training despite full training’s overfitting.
G.4. Additional Qualitative Analysis
Class-level analyses show that Gradient Blending benefits most from audio-relevant categories and avoids some failures of naive audio-visual training. Its gains over naive fusion are larger than its drops in the compared classes.
- G-Blend Versus RGB: Classes improved by G-Blend over RGB are generally audio-relevant, while dropped classes are not strongly related to audio semantics.For many dropped classes, the audio model’s performance is almost 0.
- Naive Audio-Visual Versus RGB: Naive audio-visual training improves fewer top classes and drops more bottom classes than G-Blend, sometimes performing worse than the audio-only model.The cited examples include snorkeling and feeding bird, where relevant sounds are present.
- G-Blend Versus Naive Audio-Visual: G-Blend’s gains over naive audio-visual training are much larger than its decreases in the compared top-20 and bottom-20 classes.