Source-linked AI summary
Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
Itai Gat, Idan Schwartz, Alexander Schwing, Tamir Hazan
TL;DR
Multi-modal classifiers can favor a subset of modalities, leaving other inputs less used. The paper regularizes functional entropy through a log-Sobolev bound using functional Fisher information. It reports state-of-the-art results on VQA-CPv2 and SocialIQ while more uniformly exploiting modalities, and demonstrates efficacy on Colored MNIST.
Problem
Method
Results
Takeaways & Limitations
The regularization encourages models to exploit available modalities more uniformly.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Many recent datasets contain a variety of different data modalities, for instance, image, question, and answer data in visual question answering (VQA). When training deep net classifiers on those multi-modal datasets, the modalities get exploited at different scales, i.e., some modalities can more easily contribute to the classification results than others. This is suboptimal because the classifier is inherently biased towards a subset of the modalities. To alleviate this shortcoming, we propose a novel regularization term based on the functional entropy. Intuitively, this term encourages to balance the contribution of each modality to the classification result. However, regularization with the functional entropy is challenging. To address this, we develop a method based on the log-Sobolev inequality, which bounds the functional entropy with the functional-Fisher-information. Intuitively, this maximizes the amount of information that the modalities contribute. On the two challenging multi-modal datasets VQA-CPv2 and SocialIQ, we obtain state-of-the-art results while more uniformly exploiting the modalities. In addition, we demonstrate the efficacy of our method on Colored MNIST.
1 Introduction
Multi-modal classifiers can favor an easy modality, such as language in VQA, and ignore other useful inputs. The paper regularizes functional entropy through a functional-Fisher-information bound; it reports state-of-the-art results on VQA-CPv2 and SocialIQ with more uniform modality use.
1 Introduction
- Classical regularization can bias multi-modal classifiers toward a single, simple modality, such as language priors in VQA or color rather than shape in Colored MNIST.The paper gives answering “how many...?” with “2” as an example of a language prior.Source: §1 · p. 1
- The proposed functional-entropy regularizer encourages more balanced contributions from modalities, with the log-Sobolev inequality bounding functional entropy using functional Fisher information.Figure 1 describes measuring modality sensitivity by perturbing each modality and incorporating functional Fisher information into the loss.Source: §1 · p. 1, Figure 1 · p. 2
- The method is reported to permit state-of-the-art performance on SocialIQ (68.53% vs. 64.82%) and VQA-CPv2 (54.55% vs. 52.05%), with empirical verification on synthetic Colored MNIST.Source: §1 · p. 2
2 Related Work
Related work studies multi-modal datasets and methods for addressing dataset bias, alongside information-theoretic approaches in deep nets. This paper distinguishes its use of functional entropy and functional Fisher information from probability-distribution-based entropy and Fisher information.
Multi-modal datasets
- Multi-modal models may rely more on one modality, such as question type in VQA, despite tasks treating all modalities as essential.Prior work includes large multi-modal datasets and models, with studies identifying modality imbalance.Source: §2 · p. 2
Bias in datasets
- Bias-focused datasets test generalization across different train and test distributions, while prior approaches include regularization, resampling, adversarial representations, and bias-only models.Examples include VQA-CP and Colored MNIST.Source: §2 · p. 2
Entropy and information in deep nets
- Unlike entropy defined for probability distributions, functional entropy is defined for any non-negative function; this work uses functional Fisher information and a log-Sobolev inequality for that setting.The passage distinguishes this functional Fisher information from Fisher information defined over probability density functions.Source: §2 · pp. 2–3
3 Background
The background frames multi-modal learning as prediction from data composed of distinct modalities. It introduces functional entropy, its bound through functional Fisher information, and tensorization across product spaces.
3 Background
- In multi-modal discriminative learning, an instance can comprise multiple modalities, such as an image’s color and shape or VQA’s visual, question, and answer inputs.The model maps data instances to labels.Source: §3 · p. 3
3.1 Functional entropy
- Functional entropy is defined for a non-negative function over a continuous variable and probability measure, is non-negative, and equals zero only for a constant function.Unlike differential entropy, it need not be defined only for a normalized probability density.Source: §3.1 · p. 3
3.2 Functional Fisher information
- Because functional entropy is hard to estimate directly, the log-Sobolev inequality bounds it using functional Fisher information, which is defined using the gradient norm.The passage specifies the inequality for Gaussian measures.Source: §3.2 · pp. 3–4
3.3 Tensorization and multi-modal data
- Tensorization relates overall functional entropy on a product space to modality-level functional entropies, enabling bounds based on the separate modalities.The product measure is formed from the measures for the individual modalities.Source: §3.3 · p. 4
4 Regularization by Maximizing Functional Entropies
The method defines functional entropy around a classifier’s softmax predictions under Gaussian input perturbations, then uses a functional-Fisher-information bound to regularize training. Tensorization approximates the contribution of individual modalities by perturbing one modality at a time.
4 Regularization by Maximizing Functional Entropies
- For each training point, the method defines functional entropy over the softmax function using cross-entropy between predictions at a perturbed input and at the original input.The perturbation is Gaussian with expected value equal to the training input.Source: §4 · p. 4
- The log-Sobolev bound supplies functional Fisher information for regularization; training minimizes loss while using inverse information, with λ balancing the two.The objective is described as implicitly encouraging information from each modality.Source: §4 · pp. 4–5
4.1 Tensorization
- Tensorization approximates each modality’s contribution by varying that modality while holding the others at their training-point values, then applies the bound to regularize training.The resulting objective uses the tensorized functional-Fisher-information bound.Source: §4.1 · p. 5
5 Connection Between Functional Entropy and Variance
The section connects functional entropy to variance and develops a variance-based regularizer using the Poincaré inequality. It adapts the bound to multimodal inputs through tensorization.
5 Connection Between Functional Entropy and Variance
- For small non-negative f(z), a Taylor expansion of 1 + f(z) relates functional entropy to variance, with a non-negative residual that vanishes faster than its argument.Source: §5 · p. 5
- The Poincaré inequality gives a variance bound analogous to the log-Sobolev bound, and the entropy–variance relation suggests the bounds behave similarly in practice.Source: §5 · p. 5
5.1 Regularization using Variance
- For each training point, the variance regularizer bounds the variance of the cross-entropy between predictions on a perturbed input and the original input using the Poincaré inequality.Source: §5.1 · p. 6
- The variance bound is adapted to multimodal inputs through tensorization, yielding a learning objective that combines loss minimization with regularization.Source: §5.1 · p. 6, §5 · p. 6
6 Experiments
Experiments evaluate the regularization across four datasets, including a synthetic Colored MNIST setting designed to test whether classifiers exploit misleading features; the authors report improved generalization with the regularizer.
6 Experiments
- Table 1 compares four regularizers on Colored MNIST, SocialIQ, and Dogs & Cats; functional Fisher information yields a smaller maximum-to-convergence accuracy gap.The table reports maximum accuracy and accuracy after convergence.Source: Table 1 · p. 7
6.1 Colored MNIST
- On grayscale validation, the unregularized Colored MNIST model reaches 41.11% accuracy, while functional Fisher information regularization outperforms the same classifier by almost 55%.Without regularization, shape Fisher information is almost zero; with it, shape information exceeds color information.Source: §6.1 · p. 8
6.2 VQA-CPv2 / 6.3 SocialIQ
- VQA-CPv2 reaches 54.55% accuracy (+2.5% over the previous state of the art); on SocialIQ, validation accuracy rises from ~64% to 67.93%.The VQA-CPv2 result adds functional Fisher information regularization to the current state-of-the-art method [32].Source: §6.2 · p. 8, §6.3 · p. 9
6.3 SocialIQ
- Without functional Fisher information regularization, training and validation accuracy are ~80% and ~64%; with regularization, training accuracy is 70% and validation accuracy is 67.93%.Without regularization, the answer modality has the highest information value while the question modality is almost entirely ignored; regularization balances information between modalities.Source: §6.3 · p. 9
6.4 Dogs and Cats
- On Dogs & Cats, functional Fisher information regularization reaches 94.71% for TB1→TB2 and 88.11% for TB2→TB1, exceeding [6] by 3.5% and 1%, respectively.The TB1→TB2 result uses λ = 3e-10.Source: §6.4 · p. 9
7 Conclusion
Classical regularizers can lead multi-modal models to ignore modalities; functional-entropy regularization encourages more uniform use of the available modalities.
Broader Impact
The authors present functional-entropy regularization as a way to help classifiers benefit more uniformly from dataset modalities and reduce biases tied to modalities that are easier to leverage. They frame this work as a first step toward more inclusive machine learning.
- The authors describe more uniform use of available modalities as a way to help reduce biases in multimodal classifiers.Source: Broader Impact · p. 9
- The authors identify minority disadvantage from AI systems as a societal concern motivating this work.Source: Broader Impact · p. 9
- They illustrate potential societal relevance with colored MNIST, where their regularization reduces the color-modality effect.Source: Broader Impact · p. 9
- They also cite virtual assistants giving less helpful replies when a speaker's pronunciation is not mainstream.Source: Broader Impact · p. 9
- They characterize the research as a first step toward more inclusive machine learning.Source: Broader Impact · p. 9