PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency

arXiv:2302.12247v5cs.LGcs.AIcs.CLcs.CVcs.ITmath.IT

TL;DR

Multimodal research still faces the questions of how to quantify task-relevant interactions between modalities and which models capture them. This paper defines information-theoretic measures of redundancy, uniqueness, and synergy and develops scalable estimators. Experiments demonstrate their utility for characterizing datasets and models and for selecting models across benchmarks and case studies.

  • Problem

    How to quantify the modality interactions needed for a multimodal task and identify models suited to capture them remains an open research question.

    Source: §1 · p. 1

  • Method

    The paper defines task-related redundancy, uniqueness, and synergy measures and proposes two scalable estimators for high-dimensional multimodal data.

    Source: §1 · pp. 1–2

  • Results

    Across datasets and models, PID characterizes interaction patterns and supports successful model selection on existing benchmarks and new case studies.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    PID provides a principled approach to choosing multimodal models for a task based on its interaction profile.

    Source: §1 · p. 2

    The estimators approximate real interactions, with cluster preprocessing or unimodal models introducing optimization and generalization errors.

    Source: §5 · p. 10

Abstract

from arXiv · show

The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.

1 Introduction

The paper introduces an information-theoretic framework for quantifying modality interactions relevant to downstream tasks and for identifying models suited to capturing them. It proposes scalable PID estimators, validates them on synthetic and real-world data, and applies them to dataset and model analysis and model selection.

1 Introduction

  • The framework defines PID statistics for task-related redundancy, modality-specific uniqueness, and synergy: information shared across modalities, present in only one, or newly emerging from their combination.It brings together Partial Information Decomposition in information theory and multimodal machine learning [13] [112] [64].Source: §1 · p. 1
  • Two estimators scale PID estimation to high-dimensional data: exact convex optimization for discrete-support features and sampling-based approximation for large discrete or continuous supports.Source: §1 · p. 2
  • PID estimation is validated on synthetic datasets with known statistics and real-world data by comparing estimates with human annotations.Source: §1 · p. 2
  • Dataset quantification applies PID to large-scale multimodal datasets, matching common intuitions for language, vision, and audio while yielding insights in healthcare, HCI, and robotics.Source: §1 · p. 2
  • Across model families, PID quantifies model predictions and reveals consistent patterns in the interactions captured by different models.Source: §1 · p. 2
  • PID estimates support model selection on existing benchmarks and new expert-engaged case studies in computational pathology, mood prediction, and robotics.The aim is to infer a priori which model types suit a new multimodal task.Source: §1 · p. 2

2 Background and Related Work

The paper frames multimodal interaction analysis as estimating redundancy, unique information, and synergy between two features and a task. It motivates PID as a decomposition that distinguishes these quantities where existing interaction measures can conflate them.

2 Background and Related Work

  • The framework estimates redundant, unique, and synergistic interactions from a distribution or samples over two features and a task label.The setting is a joint distribution p(x1, x2, y), including datasets or model predictions.Source: §2 · p. 2

2.1 Partial Information Decomposition

  • PID decomposes total information about task Y into redundancy, uniqueness in each feature, and synergy that emerges only when both features are present.Its definition optimizes over distributions preserving each feature–task marginal while relaxing their coupling, so redundancy and uniqueness depend on those marginals and synergy on the full joint distribution.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2 Related Frameworks for Feature Interactions

  • Earlier information-theoretic measures can confound redundancy and synergy because three-way mutual information may be positive or negative; PID separates them as R − S [13].This distinction relates mediation to R > S and moderation to S > R [10, 36], and reveals cases where positive redundancy and synergy cancel despite I(X1; X2; Y) = 0.Source: §2.2 · p. 3
  • Prior work measures interactions with statistical tests, logits, and human annotations, or with model-based interaction scores and visualizations, but these approaches do not provide PID’s standardized decomposition.Model-based approaches include non-additive interaction measures and Shapley values, alongside qualitative tools such as DIME, M2Lens, and MultiViz.Source: §2.2 · p. 3

3 Scalable Estimators for PID

The section introduces two scalable PID estimators: exact convex optimization for smaller discrete supports and a sampling-based neural approximation for large or continuous supports. Together, they extend PID estimation to high-dimensional multimodal distributions.

3 Scalable Estimators for PID

  • CVX computes PID by convex optimization and scales to problems where |Xi| and |Y| are around 100; BATCH approximates PID for large or continuous supports.Source: §3 · p. 3

3.1 CVX: Dataset-level Optimization

  • CVX optimizes the maximum-entropy distribution matching the p(xi, y) marginals, which Bertschinger et al. [13] show equivalently solves the PID definitions.Source: §3.1 · p. 3
  • Rewriting conditional entropy as log |Y| − KL divergence yields a convex objective, while linear constraints enforce the marginal matches for the optimizing tensor Q.Source: §3.1 · pp. 3–4
  • For continuous inputs, CVX histograms each Xi to approximate the continuous joint density with a discrete distribution of finite support.Source: §3.1 · p. 4

3.2 BATCH: Batch-level Amortization

  • BATCH targets high-dimensional continuous Xi with finite Y, parameterizing a joint distribution using neural modality encoders and an outer product.Source: §3.2 · p. 4
  • BATCH enforces marginal constraints with differentiable Sinkhorn normalization, using separately trained classifiers to estimate p(y|xi), then optimizes Iq(X1; X2; Y).Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • To scale beyond a storable full joint distribution, BATCH performs optimization on m ≪ n sampled datapoints per iteration and extracts PID from the converged distribution.Source: §3.2 · p. 5

4 Evaluation and Applications of PID in Multimodal Learning

Across synthetic and real-world evaluations, PID estimators recover expected interactions, align with human judgments, characterize interactions captured by models, and support model selection and real-world applications.

4.1 Validating PID Estimates on Synthetic Data

  • Both estimators exactly recovered bitwise ground-truth PID, while CVX and BATCH agreed on predominant interaction types in high-dimensional synthetic datasets.For Gaussian mixtures, Cartesian-coordinate synergy and redundancy peaked at π/4, while the Polar representation had zero redundancy; the results illustrate that representation can change PID values despite equal total information. Noise also produced nonzero synergy in some datasets whose labels did not depend on synergy.Source: §4.1 · pp. 5–6, Table 1 · p. 5, Table 2 · p. 6

4.2 Quantifying Real-world Multimodal Benchmarks

  • Estimated interactions aligned with human judgments overall, with high inter-annotator agreement, while multimodal benchmarks showed distinct interaction patterns across tasks.MUSTARD sarcasm detection showed high synergy, and uniqueness tracked unimodal performance; VQA 2.0 and CLEVR had high synergy, with CLEVR’s synergy proportion exceeding VQA 2.0’s (83% versus 75%). Human ratings matched the highest estimated interaction in four datasets; UR-FUNNY and MUSTARD matched the second-highest, amid annotator subjectivity about uniqueness versus synergy.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Quantifying Multimodal Model Predictions

  • Synergy was hardest for models to capture (average S = 0.21 ± 0.10), while redundancy and uniqueness were captured more successfully by methods designed for those interactions.Redundancy-specialized datasets averaged R = 0.41 ± 0.11 and accuracy of 73.0 ± 2.0%; uniqueness-focused methods averaged U = 0.55 and 73.0% accuracy. Missing-modality performance drops correlated with Ui at ρ = 0.8, but when Ui ≤0.05, synergy correlated with drops at ρ = 0.73, unlike redundancy (ρ = 0.01).Source: Table 4 · p. 8, §4.3 · p. 8, Figure 4 · p. 8

4.4 PID Agreement and Model Selection

  • The top three selected models reached 95% −100% of the best model’s accuracy across 11 unseen datasets, exceeding 98.5% except MUSTARD (95.2%).Selection matched each new dataset to the most similar of 10 synthetic datasets by normalized PID values, then recommended its top three models. PID agreement correlated with accuracy at ρ = 0.81 across 10 synthetic datasets.Source: Table 5 · p. 9, §4.4 · p. 9

4.5 Real-world Applications

  • PID distinguished which modalities and interactions mattered across pathology, adolescent mood prediction, and robotic perception, and its recommendations matched observed model performance.In TCGA-LGG, genomic uniqueness was U2 = 0.06, while TCGA-PAAD showed synergy S = 0.15 exceeding U1 = 0.06 and U2 = 0.08; this agrees with Chen et al. on genomic-model performance in LGG. For MAPS text-keystroke mood prediction, synergy was 0.40 and PID-based selection identified the top three models; in robotic perception, BATCH found image uniqueness U1 = 1.79, and both estimators implied little benefit from multimodal modeling.Source: §4.5 · p. 10

5 Conclusion … B.2 Alternate Methods for Working with Continuous X1, X2:

The paper develops scalable estimators for quantifying multimodal redundancy, uniqueness, and synergy, and demonstrates their use for dataset and model analysis and selection. It also describes limitations in estimating real-world interactions and outlines future directions for PID-based learning.

5 Conclusion

  • The framework proposes scalable estimators for redundancy, uniqueness, and synergy, and demonstrates their utility for dataset quantification, model quantification, and model selection.The authors note that estimates can be affected by preprocessing or unimodal-model errors, some datasets make interaction types hard to distinguish, and real-world data lack known generation processes.Source: §5 · p. 10
  • The authors identify limits from estimator approximation, datasets with hard-to-separate interaction types, and reliance on human judgment or downstream measures for real-world data.They expect advances in density estimation, generative models, and unimodal classifiers to address some estimation errors.Source: §5 · p. 10
  • Future work includes PID-targeted dataset creation, PID-optimized representation learning, higher-dimensional information theory, and privacy-preserving and fair representation learning.The paper notes that guarantees for fairness and privacy-preserving learning could be particularly impactful.Source: §5 · pp. 10–11

A Partial Information Decomposition

  • PID decomposes I(X1, X2; Y) into redundancy, unique information from each input, and synergy.The framework defines synergy by comparing total multimodal information in the observed distribution with that in a marginal-matching, non-synergistic distribution.Source: A Partial Information Decomposition · p. 18

B Scalable PID Estimators

  • The scalable-estimator section presents algorithmic and implementation details for estimating PID values on high-dimensional continuous datasets, including CVX for discrete representations.CVX constrains representation size through histogramming or clustering, then uses a convex reformulation and conic solvers to obtain q* and PID values.Source: B Scalable PID Estimators · p. 19, §B.1 · p. 19, Figure 8 · p. 20

B.1 Estimator 1: CVX for Discrete Representations

  • CVX is an exact convex-optimization estimator that scales to problems with |Xi| and |Y| around 100.The implementation uses CVXPY and conic solvers; the authors report MOSEK as fastest and most stable, while ECOS and SCS work for smaller problems.Source: §B.1 · p. 19, Figure 8 · p. 20
  • When empirical frequency probabilities become too small, ill-conditioned optimization can cause all three conic solvers to crash; the paper reports this at sample counts such as ≥1e8.The issue can arise when a bin contains only one sample.Source: §B.1 · p. 19

B.2 Alternate Methods for Working with Continuous X1, X2:

  • Nearest-neighbor discretization performs density estimation but is ill-suited to approximating q*, while Pakman et al. [80] optimize continuous q* using Gaussian copulas.The cited approach restricts itself to Gaussian copulas because optimizing across possible joint distributions is nontrivial.Source: §B.2 · p. 20

B.3 Estimator 2: BATCH for Continuous Representations · C Experiments · C.1 Synthetic Bitwise Features

The paper presents BATCH as a scalable PID estimator for high-dimensional continuous inputs and reports that both proposed estimators exactly recover ground-truth PID values on synthetic OR, AND, and XOR interactions. It also describes limitations involving model expressivity and batch-based alignment, and notes that synergy may be underestimated.

B.3 Estimator 2: BATCH for Continuous Representations

  • BATCH targets high-dimensional continuous inputs with an end-to-end model that estimates PID from sampled data.It parameterizes a joint distribution and optimizes a training objective; the input and output supports are treated as infinite and non-enumerable.Source: §B.3 · p. 20
  • BATCH represents the joint distribution with a learned similarity matrix, projects it to satisfy marginals using Sinkhorn normalization, then optimizes the objective and extracts PID.The overview describes mini-batches as approximating full-batch optimization and the PID extraction as sampling from the learned distribution.Source: Figure 9 · p. 21, §B.3 · p. 21
  • The Sinkhorn projection uses frozen unimodal estimates of p(y|xi) to satisfy the learned joint distribution’s marginal constraints.The marginals are expanded as p(xi)p(y|xi), with p(xi) supplied by dataset samples.Source: §B.3 · pp. 20–21
  • An insufficiently expressive multimodal model can underestimate synergy, while leaving the other PID values unaffected by that model choice.The paper recommends investigating more complex models until estimated synergy stops increasing; it reports BATCH synergy is generally larger than CVX synergy.Source: §B.3 · p. 22
  • BATCH’s learned distribution is limited by network expressivity and within-batch alignments, while PID estimates are reported as insensitive to hyperparameters after convergence.The paper uses a batch size of 256 to mitigate the alignment limitation.Source: §B.3 · pp. 22–23
  • For Cartesian-coordinate GMMs, PID-component trends are similar across mean-vector angles, but synergy and redundancy contributions vary with mean-vector norm, especially for hard labels.The figure varies the norm across 0.1, 0.5, 1.0, 1.5, and 2.0 and contrasts soft and hard labels.Source: Figure 10 · p. 23

C.1 Synthetic Bitwise Features

  • Both estimators exactly recover the ground-truth PID values for synthetic OR, AND, and XOR interactions.The ground truth is attributed to Bertschinger et al. [13].Source: §C.1 · p. 23

C.2 Gaussian Mixture Models (GMM) · C.3 Synthetic Generative Model

The GMM experiments show how PID components vary with coordinate system, centroid angle, and separation, while a synthetic generative model tests whether estimators recover designed modality interactions. The synthetic results show agreement between CVX and BATCH overall, but also reveal spurious synergy when total mutual information is small.

C.2 Gaussian Mixture Models (GMM)

  • GMM total information is independent of centroid angle, increases with centroid separation, and approaches log(2) as the clusters become distinguishable.For Cartesian coordinates, unique information dominates near angles 0 and π/2, while redundancy and synergy peak at π/4.Source: §C.2 · p. 23
  • With nonnegative-radius Polar coordinates, PID values are independent of centroid angle and unique information from θ dominates.For hard labels the optimal classifier depends only on θ; for soft labels, θ's dominance sharpens as centroid separation increases, while r can provide synergy when centroids are close.Source: §C.2 · pp. 24–25, Figure 12 · p. 24
  • For signed-radius Polar coordinates, redundancy is zero, synergy appears independent of centroid separation, and redundancy grows relative to synergy as separation increases.Hard labels contain 0.65 nats of total information, near log(2), with a 0.04-nat deficit attributed to discretization; soft-label information approaches log(2) as separation grows.Source: §C.2 · p. 24, Figure 11 · p. 24
  • In Cartesian GMMs, the estimated q* becomes increasingly skewed and thin as the centroid angle grows, and its objective diverges at 45 degrees.The authors hypothesize that the continuous optimum is a degenerate one-dimensional GMM with mass only along q(x, x).Source: §C.2 · p. 25

C.3 Synthetic Generative Model

  • The synthetic model maps shared and modality-specific latent concepts through fixed transformations into 100-dimensional modalities, then generates labels from redundant, unique, synergistic, or mixed concepts.It adds nonlinear transformations, dropout, and random noise; ten datasets include four interaction-specific cases and mixed cases.Source: §C.3 · pp. 25–26
  • The synthetic ground-truth interactions are derived by bounding input-label mutual information from test accuracy, then allocating it among the interactions used to generate each label.Test accuracies of 67–75% correspond to 0.42–0.59 bits of total mutual information.Source: §C.3 · p. 26
  • CVX and BATCH agree with the designed interactions overall, but CVX reports spurious synergy in synergy-free datasets when total mutual information is small.BATCH generally estimates a larger sum of PID values, attributed to information lost during CVX clustering.Source: §C.3 · p. 26

C.4 Quantifying Multimodal Datasets

The section quantifies multimodal interactions across real-world datasets, finding that interaction patterns differ by dataset and that QA datasets are especially synergy-dominant. Comparisons of CVX, BATCH, and human judgments show agreement on relative interaction patterns, though estimated magnitudes differ.

Results on multimodal fusion

  • Across multimodal fusion datasets, interaction patterns vary: sarcasm shows high synergy, uniqueness tracks unimodal performance, and high- versus low-synergy datasets align with benefits or lack of benefit from interaction modeling.The full results are reported in Table 6.Source: Table 6 · p. 27, §C.4 · p. 27

Setup

  • MultiBench spans 10 modalities, 15 prediction tasks, and 5 research areas, testing feature learning and complex real-world interactions between modalities and labels.The section also includes Visual Question Answering [39] and CLEVR [53] to test language grounding in the visual domain.Source: §C.4 · p. 27

Estimator details

  • CVX applies PCA and clusters each modality into 20 groups, whereas BATCH estimates PID from pretrained image and text features and unimodal and multimodal classifiers.BATCH uses CLIP-VIT-B/32 [86] encoders and 2-layer MLP classifiers; CVX estimates each modality pair separately for three-modality datasets.Source: §C.4 · p. 27

Results on QA

  • Synergy is consistently the largest interaction on QA datasets, consistent with VQA 2.0 balancing yes/no answers across different images for the same question.The paper relates this pattern to prior work on dataset balancing and models trained on these datasets [39].Source: Table 7 · p. 28, §C.4 · p. 28

Comparing CVX and BATCH

  • On VQA and CLEVR, CVX, BATCH, and human judgments agree on the largest interaction and relative ordering, while their absolute interaction values can differ.The authors attribute magnitude differences to information loss in CVX clustering and BATCH representation learning, and recommend relative comparisons to identify the dominant interaction.Source: Table 7 · p. 28, §C.4 · p. 28

C.5 Comparisons with Human Judgment · C.6 Comparisons with Other Interaction Measures

Human annotations provide a reliable but sometimes subjective account of multimodal interactions, while CVX and BATCH estimates on synthetic datasets align with ground-truth interactions. The comparisons also distinguish data-level interaction measures from methods that quantify interactions captured by trained models.

C.5 Comparisons with Human Judgment

  • Annotators rated redundancy, each modality’s unique information, and synergy from 0–5 after viewing both modalities and the task label.They also rated confidence on a 0–5 scale.Source: §C.5 · p. 28
  • Human annotations show quite high inter-annotator agreement and confidence, indicating that the annotated interaction results are reliable.Table 9 reports annotator agreement and average confidence scores for real-world multimodal datasets.Source: §C.5 · p. 29, Table 9 · p. 29
  • Annotations matched prior intuitions: VQA and CLEVR showed high synergy, while language had high U1 in sentiment, humor, and sarcasm.Source: §C.5 · p. 29
  • Agreement on U1 and S was low for MOSEI (−0.09, −0.07), UR-FUNNY (−0.14, −0.03), and MUSTARD (−0.08, −0.04).The authors attribute this to subjectivity in deciding whether sentiment, humor, or sarcasm is present only in language or emerges when language and video are contextualized together.Source: §C.5 · p. 29
  • Agreement on synergy was 0.13 for videos with the attribute present, versus −0.10 when it was absent.This comparison came from examining attribute-present and attribute-absent videos separately.Source: §C.5 · p. 29

C.6 Comparisons with Other Interaction Measures

  • Other information-theoretic measures include shared and conditional mutual information used in co-training and multi-view learning [102], but such measures can over- or underestimate interactions or be negative.Source: §C.6 · pp. 29–30
  • On synthetic datasets, CVX and BATCH agree on relative interaction values and are consistent with ground-truth interactions.I-min can overestimate synergy and uniqueness; WMS measures synergy minus redundancy and can be negative or cancel to zero; CI can be negative and can incorrectly assign highest uniqueness to S-only data.Source: §C.6 · p. 30, Table 10 · p. 30
  • Shapley values, Integrated Gradients, and CCA quantify interactions captured by a multimodal model, whereas this work treats interactions as properties of data before model training.Source: §C.6 · p. 30

C.7 Quantifying Model Predictions · C.8 Multimodal Model Selection

The paper measures which multimodal interactions models capture and uses dataset PID profiles to guide model choice. Models specialized for redundancy, uniqueness, and synergy differ in their results, while PID-based selection performs near the best model on most evaluated datasets.

C.7 Quantifying Model Predictions

  • The model suite spans fusion levels, interaction inductive biases, and training objectives, including methods oriented toward redundancy, synergy, and uniqueness.Uniqueness-oriented implementations include lower-order terms and reconstruction objectives; PID is run on validation predictions to summarize captured interactions.Source: §C.7 · pp. 30–31
  • Model PID measures consistently exceeded dataset measures, and their sum indicated test performance because it equals I({X1, X2}; Y), the total explained mutual information.The full model results cover all models trained on all datasets; the concise overview compares each interaction type on its specialized dataset.Source: §C.7 · p. 31
  • Redundancy was captured at 0.41 ± 0.11 on average, with 73.0 ± 2.0% accuracy; additive, agreement, and alignment methods were particularly strong.Tensor-fusion methods and reconstruction objectives also captured redundancy well.Source: §C.7 · p. 31
  • Uniqueness was harder to capture than redundancy, averaging 0.37 ± 0.14, but lower-order interaction and reconstruction methods reached U = 0.55 and 73.0% accuracy.Additive and agreement methods performed poorly on uniqueness.Source: §C.7 · p. 31

C.8 Multimodal Model Selection

  • The top three chosen models all achieved 95%–100% of the best-performing model’s accuracy, and exceeded 98.5% on all datasets except MUSTARD.The pipeline’s closest-model selection yielded >96% performance compared with exhaustively trying all models.Source: §C.8 · p. 39, Figure 17 · p. 39
  • For a new dataset, the selection procedure normalizes its PID profile, finds the closest base-dataset profile by absolute difference, and evaluates that dataset’s top three models.It was simulated on five new synthetic and six real-world datasets, with selected-model accuracy compared against the best model’s accuracy on each dataset.Source: §C.8 · p. 39

C.9 Real-world Case Study 1: Computational Pathology

The case study applies PID to pathology–genomics interactions in cancer survival prognostication, finding little mutual information and that multimodal integration benefits TCGA-PAAD but not TCGA-LGG. It uses these measures to refine insights into when multimodal integration may help.

C.9 Real-world Case Study 1: Computational Pathology

  • Cancer prognosis commonly integrates whole-slide histology with molecular information, while prior work found unimodal models can be competitive for some cancers.In diffuse gliomas, histological subtypes and molecular markers inform treatment decisions; prior work suggested some cancers may be prognosticated using one modality.Source: §C.9 · p. 39
  • PID found little mutual information; synergy was S = 0.02 in TCGA-LGG and S = 0.15 in TCGA-PAAD, indicating multimodal benefit only for PAAD, consistent with c-index improvement and significant patient stratification.Uniqueness values also support genomics as the stronger source of task-specific information when c-index performance is high, and pathology when it is low.Source: Table 20 · p. 40, §C.9 · p. 40
  • The study analyzed TCGA lower-grade glioma (n = 479) and pancreatic adenocarcinoma (n = 209), using survival time binned into four categories and pathology and genomic features.It trained ABMIL and SNN unimodal models, clustered features with k = 3, and used the resulting modality-label pairs to compute PID measures.Source: §C.9 · p. 40

C.10 Real-world Case Study 2: Mood Prediction from Mobile Data · C.11 Real-world Case Study 3: Robotic Perception

The mood-prediction case finds that text paired with apps or keystrokes has high synergy, and PID-based selection closely matches the best-performing models. In robotic perception, image information is most unique, while the second modality adds little beyond redundant information.

C.10 Real-world Case Study 2: Mood Prediction from Mobile Data

  • For text + apps and text + keystrokes, estimated synergy is S = 0.26 and S = 0.4, respectively, alongside redundancy R = 0.09 and R = 0.12.The corresponding second-modality uniqueness values are U2 = 0.09 and U2 = 0.04.Source: Table 21 · p. 41, §C.10 · p. 41
  • For MAPST,A, REC (acc = 0.56%) and MULT (acc = 0.48%) lead; for MAPST,K, TENSOR (acc = 0.57%) and REC (acc = 0.52%) lead, and PID selects REC and TENSOR.Selection agreement is α = 0.66 for MAPST,A and α = 0.74 for MAPST,K, corresponding to 98.3% and 99.85% of the highest agreement, respectively.Source: Table 22 · p. 41, §C.10 · p. 41
  • The mood task uses daily self-reports binned into negative, neutral, and positive classes, with 1641 samples from 17 participants and smartphone text, keystroke, and app-use data.Participants with fewer than 50 self-reports were excluded; the three classes comprise 12.43%, 43.63%, and 43.94% of the data.Source: §C.10 · pp. 40–41

C.11 Real-world Case Study 3: Robotic Perception

  • In robotic perception, BATCH estimates image uniqueness at U1 = 1.79, while both estimators predict low U2 and some redundancy; multimodal and modality-1-only performance shows no difference.The passage notes that the image is the best unimodal predictor [62].Source: Table 23 · p. 42, §C.11 · p. 42
  • The robotic task uses 1000 simulated MuJoCo PUSH [59] trajectories to predict a puck’s 2-D pose from grayscale images, force/contact sensing, and end-effector position.CNN feature extractors and LSTMs are used; continuous labels are discretized into 10 equal-count training bins for CVX and BATCH.Source: §C.11 · p. 42
  • CVX underestimates U1 and R because its clustering loses information in high-dimensional time-series modalities.CVX clusters data before processing, making those modalities difficult to describe without information loss.Source: §C.11 · p. 42

D Summary of Takeaway Messages and Future Work · E Limitations and Broader Impact

The authors present PID as a tool for quantifying dataset and model interactions and guiding model selection, and outline future directions in pointwise measures, interaction-aware representations, and fairness or invariance.

D Summary of Takeaway Messages and Future Work

  • PID provides reliable indicators of the nature and quantity of interactions in synthetic and real-world multimodal datasets, and dataset creators should report estimated values with justification.The justification should explain whether the values are expected given how the features were selected.Source: D Summary of Takeaway Messages and… · p. 42
  • PID reasonably indicates interactions captured by trained models, though parameter counts, training trajectories, and non-convex optimization introduce noise; researchers should state their assumptions and intended interactions.The authors still observe consistent patterns in the interactions captured by different models.Source: D Summary of Takeaway Messages and… · p. 42
  • For model selection, high U1 and U2 may make unimodal models sufficient, high R suggests agreement, alignment, ensemble, or co-training, and high S favors complex-interaction multimodal methods.For high synergy, the authors suggest methods based on tensors, multiplicative interactions, and self-attention.Source: D Summary of Takeaway Messages and… · pp. 42–43
  • Pointwise interaction measures could quantify each datapoint’s contribution to redundancy, uniqueness, or synergy, supporting finer-grained analysis, error analysis, and active learning.The proposed measures would assess a datapoint in the context of the full distribution.Source: D Summary of Takeaway Messages and… · p. 43
  • Architectures or training objectives designed to capture PID-quantified synergy could improve representations; Transformers already include some notion of multiplicative interactions.The authors suggest synergy may heuristically contribute to Transformers’ success.Source: D Summary of Takeaway Messages and… · p. 43
  • PID could also support fairness, privacy, and feature-invariance objectives by removing information one variable contributes about a task in the context of another variable.The framework currently measures information that two variables contribute toward a task, but the authors suggest a converse use for removing information.Source: D Summary of Takeaway Messages and… · p. 43

E.1 Limitations of our Estimators · E.2 Broader Impact

The paper notes that its estimators have distinct limitations and that PID-based analysis should be accompanied by care in interpreting results and assessing social and environmental risks. It presents the framework as a way to study multimodal interactions while supporting work toward interpretable, lightweight models.

E.1 Limitations of our Estimators

  • CVX rarely scales beyond several thousand classes, and large |Xi| can produce near-zero empirical frequencies that destabilize conic solvers.These issues are particularly relevant in real-world applications.Source: §E.1 · p. 43
  • BATCH scales to high-dimensional continuous distributions but can incur representation and mini-batch bias, alongside neural-network convergence and tuning challenges.Potential issues include local minima and unstable convergence when learning ˜q and ˆp.Source: §E.1 · p. 43

E.2 Broader Impact

  • The framework aims to systematically quantify multimodal datasets and models, while the authors call for careful deployment and deeper engagement with domain experts.The authors say released datasets, models, and code can support scalable future quantification of feature interactions and that PID can help study social and environmental issues and enable interpretable, lightweight models.Source: §E.2 · pp. 43–44
  • PID values are approximate, so users should avoid overfitting to them and use other visualization and quantification methods for a holistic dataset understanding.The paper frames this caution as applying to approximate estimators generally.Source: §E.2 · p. 43
  • Multimodal predictions can pose privacy risks for human video or health data; the real-world healthcare and affective-computing datasets used public data released with participant or creator consent.The authors also acknowledge exposure bias from imbalanced datasets, which can bias models toward particular genders, races, and demographic groups.Source: §E.2 · pp. 43–44
  • Internet-scale models can impose hardware, electricity, computation, and carbon costs, motivating future study of model size and learned interactions.Preliminary experiments found that smaller models could still capture high degrees of each interaction, potentially informing inductive biases that use fewer parameters.Source: §E.2 · p. 44

Related papers

Loading 2302.12247v5…