Source-linked AI summary
Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, Longbo Huang
TL;DR
The paper addresses the limited theoretical understanding of why jointly trained multi-modal networks can underperform the best uni-modal network. Using a simplified data distribution and late-fusion network trained by gradient descent, it proves that modality competition causes only a subset of encoders to learn useful representations, producing a performance gap supported by experiments.
Problem
The paper asks why the best uni-modal network can outperform a jointly trained multi-modal network despite multi-modal learning's practical success.
Method
The paper analyzes late-fusion multi-modal and uni-modal networks with smoothed ReLU under gradient descent on a sparse-coding data model containing insufficient modality data.
Results
The analysis proves modality competition: random initialization causes only a subset of modality encoders to learn sufficient representations, establishing a test-error gap between uni-modal and jointly trained multi-modal networks.
Takeaways & Limitations
The losing modalities are identified as the source of joint training's sub-optimality, and experiments show that this competition matches late-fusion joint-training behavior.
Takeaways & Limitations
The theory uses a simplified data distribution, including unitary dictionaries with orthogonal columns and insufficient-modality structure.
Abstract
from arXiv · showhide
Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the jointly trained multi-modal network, which is counter-intuitive since multiple signals generally bring more information. This work provides a theoretical explanation for the emergence of such performance gap in neural networks for the prevalent joint training framework. Based on a simplified data distribution that captures the realistic property of multi-modal data, we prove that for the multi-modal late-fusion network with (smoothed) ReLU activation trained jointly by gradient descent, different modalities will compete with each other. The encoder networks will learn only a subset of modalities. We refer to this phenomenon as modality competition. The losing modalities, which fail to be discovered, are the origins where the sub-optimality of joint training comes from. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.
1 Introduction
The paper studies why jointly trained multi-modal networks can underperform the best uni-modal network, focusing on what happens during neural-network training. It theoretically and empirically attributes this gap to modality competition, where joint training learns only a subset of modality representations.
- The paper asks how modality encoders learn representations during multi-modal training and why naive joint training can fail in deep learning.
- The analysis uses a simplified multi-modal data model with insufficient data, representing cases where one modality alone cannot adequately predict the task.
- Uni-modal training focuses on modality-associated features and yields good performance, according to Theorem 5.1.
- Naive joint training learns sufficient feature representations in only a subset of modality encoders, a phenomenon called modality competition.
- The theoretical analysis establishes a test-error performance gap between uni-modal and jointly trained multi-modal networks.
- Experiments on item classification compare image-only, text-only, and jointly trained text-plus-image models, supporting the predicted superiority of the best uni-modal network.
2 Related Work
Prior work established the practical success of multi-modal learning and developed theoretical analyses, but existing theory often assumes each modality is individually sufficient. This paper focuses on feature learning and training dynamics under more realistic insufficient-modality settings.
- Multi-modal learning has been applied to speech recognition, semantic segmentation, VQA, sound localization, and related tasks.
- Existing multi-view analyses typically assume that each view alone is sufficient for accurate prediction, unlike settings where individual modalities can be incomplete.
- Recent theory studies generalization, performance drops, and neural-network feature learning, but does not fully analyze the training process addressed here.
3 Notations
The paper defines notation for index sets, vector sparsity, norms, asymptotic rates, high-probability statements, and random-variable support.
- [K] denotes the index set {1, . . . , K}.
- For vectors, ∥x∥0 counts nonzero elements and ∥x∥∞ is the maximum absolute coordinate.
- The paper uses standard big-O variants and e O(·) to hide polylog(K) factors.
- w.h.p. means with probability at least 1 −e−Ω(log2(K)), while Supp(·) denotes random-variable support.
4 Problem Setup
The paper models two-modality classification with sparse coded inputs, sufficient and insufficient modality instances, and Gaussian or feature noise. It analyzes late-fusion networks with smoothed ReLU trained by gradient descent, alongside corresponding uni-modal networks.
- Data distribution: The setup considers multi-class samples X=(X1,X2), with each modality represented by a vector generated from a sparse coding model.
- Data distribution: The sparse-coding model uses dictionaries, sparse vectors, and noise, with the main analysis focusing on unitary dictionaries with orthogonal columns.
- Data distribution: Each modality can be sufficient or insufficient; insufficient data have a relatively small target coefficient that can be concealed by off-target signal.
- Learner Network: The learner uses a two-modality late-fusion architecture whose modality features are summed and passed to a linear classifier.
- Learner Network: The modality encoders are single-layer networks with smoothed ReLU activation, and the analysis also defines corresponding uni-modal networks without fusion.
- Training: Model parameters are optimized with empirical cross-entropy loss using gradient descent on multi-modal and uni-modal training data.
5 Main Results
The paper proves that uni-modal networks can learn good modality-specific representations, while late-fusion joint training can achieve zero training error yet leave some modality encoders deficient. This modality competition produces a theoretical performance gap between the best uni-modal and jointly trained multi-modal networks.
- Uni-modal Network Results: Theorem 5.1 shows that, after sufficiently many iterations, uni-modal networks attain zero empirical training error and good test performance.The guarantee holds for sufficiently large K and learning rate η ≤ 1/poly(K).
- Uni-modal Network Results: Uni-modal test error primarily comes from insufficient data that lack enough feature-related information for classification.Here, µr denotes the proportion of data insufficient for modality Mr, supporting the interpretation that the learned representation is ideal for that modality.
- Multi-modal Network with Joint Training: Theorem 5.2 shows that joint training can reach zero training error while each modality-specific probe has a non-trivial probability of high test error.The probe fr(T) evaluates the feature representation learned by modality Mr using a fixed linear head and that modality’s input.
- Multi-modal Network with Joint Training: At least one modality encoder therefore learns deficient modality-associated features, despite the jointly trained network fitting the training set.The paper characterizes this as unsatisfactory feature representation learning and reports test error over 1/K in the unfavorable case.
- Multi-modal Network with Joint Training: Naive joint training can impair the original modalities instead of exploiting additional signals, creating a gap relative to uni-modal solutions.The resulting joint-training test error is approximately a weighted average of uni-modal errors and depends on modality factors {pr}r∈[2] and {µr}r∈[2].
6 Proof Outline
The proof decomposes joint training into two phases: random initialization creates a winning modality, then training preserves its advantage while competing modalities remain underlearned.
- Phase 1: modality competition from random initialization: Joint training first produces modality competition: random initialization lets one modality’s neurons emerge as singletons while others are barely discovered.The paper identifies this early imbalance as the origin of naive joint training’s sub-optimality.
- Phase 1: modality competition from random initialization: For each class, a winning modality is selected when its initialization correlation exceeds that of the other modality.The winning modality can arise even when the other modality has a stronger marginal target signal.
- Phase 1: modality competition from random initialization: A slight initialization difference can grow dramatically because the early dynamics resemble a tensor power method.Winning-modality neurons grow toward Θ(β), while competing neurons remain near initialization scale eO(σ0).
- Phase 2: converge to the winning modality: After competition, winning-modality neurons enter the activation’s linear region while the others remain in the polynomial or negative regime.The loss can decrease substantially even though the losing modality’s feature strength remains small.
- Phase 2: converge to the winning modality: The winning modality remains dominant through convergence because the competing modality’s feature quantity does not exceed eO(σ0).The paper contrasts this with uni-modal training, where the sole predictive signal is learned without competitors.
7 Conclusions
The paper concludes that late-fusion joint training can produce modality competition and unfavorable representations despite achieving zero training error, explaining observed test-performance gaps.
- Empirical support: Experiments show that only a subset of modalities learns good representations, matching the predicted degradation relative to directly trained uni-modal networks.The paper reports class-level accuracy drops and degraded modality representations under joint training.
- Proof strategy: The proof analyzes neural-network gradients and training phases under global parameter assumptions, using Gaussian and tensor-power arguments.The induction hypothesis is established for all iterations under the stated learning-rate and parameter conditions.
- Winning modality: At most one modality can win for each class under the paper’s initialization-based definition.The winning modality is determined at iteration t = 0.
A.5.1 Error for Insufficient Data
The insufficient-data analysis bounds classification errors during training using gradient, noise-correlation, and phase-based arguments under the induction hypothesis.
- Error analysis: The analysis assumes the induction hypothesis and bounds errors for insufficient data across training iterations.It separately tracks phase intervals and gradient contributions for insufficient examples.
- Error analysis: For insufficient data, the correct-label loss contribution becomes exponentially small in polylog(K) under the stated bounds.The passage gives 1 − ℓy(F(t), X) ≤ exp(−polylog(K)).
- Proof structure: The proof decomposes the later training interval into 2m + 1 subintervals to control the evolving error.This decomposition is used after the network enters the final convergence analysis.
A.5.2 Error for Sufficient Data
The sufficient-data analysis shows how learned modality correlations control prediction errors across the two training phases and establish the persistence of winning modalities.
- Training phases: For sufficient data, the proof tracks individual errors and separates the regime before and after the network’s output becomes significant.The later stage begins once relevant feature quantities reach Θ(1/m).
- Activation regimes: Winning-modality pre-activations enter the activation’s linear regime for sufficient examples, while other cases are bounded separately.The analysis distinguishes sufficient from insufficient multi-modal data and tracks target-label and non-target-label cases.
- Data contributions: The proof controls the contribution of insufficient data as negligible in the early phase relative to sufficient multi-modal data.This simplification is used when the insufficient-data count is much smaller than the total sample count.
- Winning-modality persistence: Once the winning modality’s feature quantity reaches eΩ(1), the competing modality remains bounded and cannot overtake it.This bound is used to show that the winner persists through training.
- Theorem closure: The induction hypothesis is proved for all iterations under the global parameter settings and sufficiently large K.The theorem requires η ≤ 1/poly(K).
A.9 Main Theorems for Multi-mdoal
The multi-modal joint-training analysis proves that modalities compete during learning, allowing only some modalities to be discovered. Despite sufficiently small training error, the resulting network can retain high test error.
- Main theorem: Joint training yields modality competition, so the learned encoders discover only a subset of modalities.The proof analyzes winning modalities and distinguishes sufficient from insufficient modality data.
- Training error: Training error is zero by the end of the analyzed iteration T.The proof bounds the training objective throughout Phase 2, Stage 2 using gradient descent's non-increasing behavior.
- Test error: For some modalities, the test error is high with probability determined by the competing modality's winning probability.The analysis attributes the main test error to insufficient data and gives corresponding probability terms p_j,r.
B.2 Main theorem for Uni-modal
The uni-modal analysis proves learning guarantees for each modality after polynomially many iterations. The learned uni-modal network has zero training error, while its test error is characterized separately for sufficient and insufficient modality data.
- Main theorem: After T = poly(k) iterations, each learned uni-modal network satisfies the stated high-probability guarantees.The theorem assumes sufficiently large K and η ≤ 1/poly(k).
- Training error: The learned uni-modal network achieves zero training error.The proof states that the training error becomes sufficiently small at the end of iteration T.
- Test error: For sufficient modality data, the test analysis bounds the correct-class output and compares competing class coordinates.The proof uses function approximation and the data distribution to derive the test-error characterization.
- Test error: Insufficient modality data occurs with probability µr and contributes to the uni-modal test-error analysis.The proof separately considers insufficient data and records its occurrence probability.
C Experimental Setup
The experiments use an internal product-classification dataset containing image and title-text modalities. They evaluate Transformer-based image and text models trained end to end with AdamW.
- Dataset: The internal dataset contains about 600K training samples and 10K validation samples.Training accuracy is evaluated on a separate sample of 10K training products.
- Dataset: Each product pairs an image with title text describing information such as category and features.The image is typically a product photograph.
- Model: The image and text modalities use separate Transformer models, including a small six-layer ViT image encoder.The image input is resized to 256 × 256 and divided into 16 × 16 patches.
- Evaluation: The study also compares a single-modal model trained independently with one using a frozen encoder initialized from the corresponding multi-modal model.A linear classifier is added on top of the frozen encoder in the latter setup.
- Optimization: All models are trained end to end with AdamW using a peak learning rate of 1e-4 and total batch size 256.Experiments run on 16 NVIDIA V100-32G GPUs with warmup and cosine decay.