Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks

Nan Wu; Stanisław Jastrzębski; Kyunghyun Cho; Krzysztof J. Geras

Source-linked AI summary

Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks

Nan Wu, Stanisław Jastrzębski, Kyunghyun Cho, Krzysztof J. Geras

arXiv:2202.05306v3Feb 10, 2022cs.LGcs.CV

TL;DR

Multi-modal DNNs can greedily rely on one modality while under-fitting others, limiting their generalization. The paper measures this imbalance with conditional utilization, introduces conditional learning speed as a training-time proxy, and balances it during training. The resulting algorithm improves generalization across Colored MNIST, ModelNet40, and NVGesture.

Problem
Multi-modal DNNs may rely on one modality while under-utilizing other predictive modalities, producing poor generalization.
Method
The paper measures conditional utilization, estimates it during training with conditional learning speed, and uses balanced multi-modal learning to equalize modality learning speeds.
Results
Balanced multi-modal learning improves model generalization across Colored MNIST, ModelNet40, and NVGesture.
Takeaways & Limitations
Balancing learning across modalities enables models to use all modalities appropriately and enhances generalization on the evaluated datasets.
Takeaways & Limitations
NVGesture results are not directly comparable with other works because 20% of the training samples are used for validation.

Abstract

from arXiv · show

We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend to rely on just one modality while under-fitting the other modalities. Such behavior is counter-intuitive and hurts the models' generalization, as we observe empirically. To estimate the model's dependence on each modality, we compute the gain on the accuracy when the model has access to it in addition to another modality. We refer to this gain as the conditional utilization rate. In the experiments, we consistently observe an imbalance in conditional utilization rates between modalities, across multiple tasks and architectures. Since conditional utilization rate cannot be computed efficiently during training, we introduce a proxy for it based on the pace at which the model learns from each modality, which we refer to as the conditional learning speed. We propose an algorithm to balance the conditional learning speeds between modalities during training and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm improves the model's generalization on three datasets: Colored MNIST, ModelNet40, and NVIDIA Dynamic Hand Gesture.

1. Introduction

Multi-modal DNNs can ignore useful modalities because greedy learning favors whichever modality is learned faster, producing imbalanced utilization and weaker generalization. The paper measures this imbalance and proposes balancing learning speeds to improve modality use across three datasets.

Greedy learning makes multi-modal DNNs rely on one modality while under-utilizing others, often harming generalization.
Conditional utilization rates quantify each modality’s accuracy gain when added alongside another modality.The paper reports u(depth|RGB) = 0.63 and u(RGB|depth) = 0.01 for hand-gesture recognition, indicating near-exclusive reliance on depth.
Conditional utilization is consistently imbalanced across tasks, with models often exploiting one modality and largely ignoring another.
Conditional learning speed serves as a training-time proxy for utilization, enabling balanced multi-modal learning to guide models toward underused modalities.
Balanced multi-modal learning improves modality use and generalization on Colored MNIST, ModelNet40, and NVGesture.

2. Related Work

Prior work reports that multi-modal models may exploit modality-specific biases or perform worse than strong unimodal models. This paper explains those observations through greedy learning and develops tools to address inadequate modality utilization.

Earlier studies found multi-modal models exploiting modality-wise bias in VQA and sometimes underperforming unimodal models in video-related tasks.
The paper attributes inadequate modality utilization across these settings to the greedy nature of multi-modal learning.
Unlike prior work focused mainly on architecture or bias inspection, this study provides tools to diagnose and overcome inadequate modality utilization.

3. Problem Setup

The studied model uses two interconnected unimodal branches with intermediate fusion, producing branch predictions whose average is the final prediction. Training minimizes modality-specific cross-entropy losses with SGD.

The model has two unimodal branches, φ0 and φ1, connected by layer-wise intermediate-fusion modules processing modalities m0 and m1.
Fusion modules use MMTM to pool feature maps, generate channel activations, and rescale each branch’s features.
The two branch outputs, ˆy0 and ˆy1, are averaged to produce the multi-modal prediction ˆy.
Training updates the model with SGD to minimize modality-specific cross-entropy losses, selecting the checkpoint with highest validation accuracy.

4. The Greedy Learner Hypothesis

The paper defines conditional utilization to measure modality dependence and hypothesizes that greedy training favors the modality learned fastest. Experiments test this hypothesis by deriving modality-isolated models and comparing their accuracies.

4.1. Conditional Utilization Rate: The analysis derives modality-isolated variants by cutting information sharing between unimodal branches and compares their test accuracies.
4.1. Conditional Utilization Rate: Conditional utilization measures a modality’s marginal accuracy contribution given the presence of the other modality.
4.1. Conditional Utilization Rate: A large absolute utilization difference, |dutil(f)|, indicates that the model benefits from one modality but not reciprocally from the other.
4.2. Multi-modal Learning Process is Greedy: The greedy learner hypothesis states that training tends to produce models relying on the modality learned fastest while neglecting other available modalities.
4.2. Multi-modal Learning Process is Greedy: Experiments across datasets and architectures find persistently imbalanced modality utilization, while less greedy training improves performance.

5. Making Multi-modal Learning Less Greedy

The paper makes multimodal learning less greedy by measuring relative learning speeds between modalities and using them to rebalance training. Its algorithm alternates regular and targeted re-balancing steps when the speed imbalance exceeds a tolerance.

5.1. Conditional Learning Speed: Conditional learning speed provides an efficient training-time proxy for conditional utilization, which is otherwise measured after training from held-out accuracy.The proxy captures relative learning speed between modalities and is empirically intended to predict imbalanced utilization.
5.1. Conditional Learning Speed: The model partitions parameters into modality-specific branch components and fusion components contributing to each modality-conditioned function.The parameter groups include θ0, θ1 and corresponding fusion-related components θ′0 and θ′1.
5.1. Conditional Learning Speed: Conditional learning speed is defined from effective parameter updates, using gradient and parameter norms to quantify how rapidly each modality-conditioned component learns.The effective update μ(θ; i) measures change from updating parameters at step i, while normalized gradient norms account for parameter direction.
5.2. Balanced Multi-modal Learning: The algorithm balances utilization online by accelerating learning from one modality at a time, choosing the target modality from the sign of the speed difference.It records cumulative update measures, computes the speed difference, and takes a re-balancing step toward m0 or m1 when the imbalance exceeds α.
5.2. Balanced Multi-modal Learning: Re-balancing steps intentionally update one unimodal branch, with feature-map modifications applied differently depending on whether m0 or m1 is being accelerated.Training begins with regular warm-up steps, switches when |dspeed(t)| exceeds α, and uses a window of Q re-balancing steps before returning to regular training.
5.2. Balanced Multi-modal Learning: The training schedule uses an imbalance tolerance α and re-balancing window size Q to govern when targeted updates begin and how long they continue.The model performs only regular steps during the first epoch, then switches modes according to the speed imbalance.

6. Experiments and Results

Experiments across three multimodal tasks show that conventional training often favors one modality, while conditional learning speed tracks this imbalance. Rebalancing learning speeds reduces modality greediness and improves generalization.

6.2. Validating the Greedy Learner Hypothesis: Many conventionally trained models have high |dutil| across all tasks, indicating reliance on one modality and neglect of another.This pattern appears across different network architectures and tasks.
6.2. Validating the Greedy Learner Hypothesis: Conditional learning-speed distributions largely replicate conditional-utilization distributions, supporting dspeed as a proxy for modality imbalance.This correspondence supports the greedy learner hypothesis linking unequal learning rates to unequal modality reliance.
6.3. Strong Regularization Encourages Greediness: Stronger L1 regularization increases parameter sparsity and, when λ ≥10^-5, significantly increases |dutil| as λ grows.|dutil| is positively correlated with R(f), and |dspeed| follows the same trend.
6.4. Balanced Multi-modal Learning: The guided algorithm produces lower |bdutil| than vanilla training, while the random variant calibrates utilization but helps generalization less.The guided and random strategies are both less greedy than vanilla, but their generalization benefits differ.
6.4. Balanced Multi-modal Learning: Guided training improves generalization over vanilla, random, and conventional strategies in all four reported cases.The authors report this comparison across the evaluated datasets and modalities.

7. Discussion

The study argues that end-to-end multimodal DNNs often underuse available modalities and reports that balancing learning across modalities enhances generalization. NVGesture results are not directly comparable with other works because 20% of training samples were used for validation.

Balancing learning across modalities enhances generalization on three multimodal datasets.The authors present adequate modality utilization as a desirable property for multimodal learning.
End-to-end multimodal DNNs often rely on one input modality while leaving others underutilized.
The greedy learner hypothesis complements existing explanations and adds tools for analyzing multimodal learning.
NVGesture results are not directly comparable with other works because 20% of training samples were used as validation data.

A. Data Preparation

The experiments use Colored MNIST, ModelNet40, and NVGesture, with each dataset represented through two input modalities. NVGesture videos receive additional temporal sampling, padding, resizing, and cropping preparation.

The study uses Colored MNIST, ModelNet40, and NVGesture as its three multimodal datasets.
Colored MNIST and ModelNet40 use the original datasets without extra preprocessing.
NVGesture inputs are sampled as 64 consecutive frames, zero-padded when shorter, resized to 256×256, and cropped to 224×224.
Colored MNIST pairs monochromatic and grayscale images, ModelNet40 uses grayscale views from 12 viewpoints, and NVGesture uses depth and RGB channels.

B. Supplementary Figures

Supplementary figures examine how utilization and learning-speed imbalance behave under different modality inputs and training settings. They also study the effects of regularization strength and algorithm hyperparameters.

Figure 7 compares dutil and dspeed distributions for distinct and identical Colored-and-gray-MNIST modalities.The figure notes that dspeed is unbounded whereas dutil is bounded by 1.
Figure 8 plots |dutil| and |dspeed| against log(λ) for models trained with different L1-regularizer weights.When log(λ) ≥ −5, both quantities increase as λ increases.
Figure 9 examines model behavior across imbalance tolerance α and re-balancing window size Q on ModelNet40 front and rear views.The learning rate is fixed at 0.1 in both hyperparameter studies.