Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning

I. INTRODUCTION

• MI measures dependency, IB specifies what to retain under compression, and PID separates retained dependency into redundant, unique, and synergistic parts.Together, these principles motivate alignment, robust fusion, interaction typing, and a foundation-model capstone.Source: §I · pp. 1–2
• The survey derives its taxonomy from information-theoretic principles because application domains indicate where methods were validated, not the mechanisms that make them work.It argues that a domain-based taxonomy can conflate different problems in one domain and separate the same problem across domains.Source: §I · pp. 1–2
• The Generalized Multimodal Information Lagrangian places alignment, robust-fusion, and interaction-typing methods at exact or approximate parameter corners and identifies unoccupied regions as candidate method families.The survey presents this coordinate system as its unifying contribution, not merely as a taxonomy.Source: §I · p. 2
• The three principles motivate alignment, robust fusion, and interaction typing, while foundation-model scaling applies all three to pretrained representations.The progression increases resolution rather than imposing strict precedence; PID can also apply directly to raw modalities.Source: §I · pp. 2–3
• The surveyed domains, including medical imaging, robotics, recommendation, and wireless communication, serve as evidence that information-theoretic mechanisms extend beyond their initial benchmarks.The survey therefore distributes domain-specific work across its framework-based sections rather than creating a separate applications-by-domain section.Source: §I · p. 3
• The survey covers 170 works from 2018–2026 and 12 foundational references, selecting studies that formally use or operationalize information-theoretic quantities.Its corpus excludes work using information-theoretic language only informally and treats likelihood-based multimodal VAEs as adjacent to its scope.Source: §I · p. 3

II. ALIGNING REPRESENTATIONS

• Cross-modal alignment maps paired observations of the same content to comparable representation-space points, primarily using MI maximization as its training signal.IB principles address what alignment leaves behind, and PID exposes an assumption shared by alignment approaches.Source: §II · p. 3

A. The Shared Estimation Toolbox

• InfoNCE distinguishes one positive pair from N −1 negatives and provides a stable MI lower bound that became a default estimator across alignment methods.MINE instead lower-bounds MI through the Donsker–Varadhan dual of KL divergence and established MI estimation as a trainable objective.Source: A. The Shared Estimation Toolbox · p. 3
• Minimizing InfoNCE or MINE does not generally guarantee a reduction in true mutual information, which compression requires.The Contrastive Log-ratio Upper Bound addresses this limitation; RINCE offers a tunable loss for noisy correspondences that violate InfoNCE’s shared-information assumption.Source: A. The Shared Estimation Toolbox · p. 4

B. CLIP-Style Contrastive Alignment: Pure MI Maximization

• CLIP trained dual image and text encoders on 400 million web pairs with symmetric InfoNCE and matched supervised ResNet-50 on ImageNet zero-shot without labels.The passage presents this as the paradigm extended by subsequent methods, none of which adds explicit compression in this subsection.Source: B. CLIP-Style Contrastive Alignment:… · p. 4
• Under population-level realizability assumptions, global temperature-optimized InfoNCE minimizers have intrinsic dimension equal to the ideal shared representation’s dimension.The account explains why effective dimensionality saturates below embedding size and temperature converges to zero.Source: B. CLIP-Style Contrastive Alignment:… · p. 4
• Symmetric InfoNCE induces a repulsive symmetric-KL coupling between modality marginals, while optimal similarity is pointwise mutual information that may exceed a d-dimensional bilinear representation’s rank capacity.The geometric account of the modality gap assumes a large-batch population-energy limit with compactness and regularity; MI maximization also does not guarantee removal of modality-specific nuisance.Source: B. CLIP-Style Contrastive Alignment:… · p. 4

C. Adding Minimality: IB-Regularized Alignment

• Under a multi-view-redundancy or equivalent task-alignment assumption, pure cross-modal MI maximization can support task sufficiency but does not enforce minimality.InfoNCE has no incentive to discard modality-specific nuisance, motivating an IB-style compression term on top of the MI objective.Source: C. Adding Minimality: IB-Regularized… · p. 4
• IB-regularized alignment methods add compression to reduce residual modality-specific information, with reported gains over CLIP, CoOp, and MaPLe.LCIBR adds a residual-information penalty; OVA-IB generalizes to arbitrarily many modalities and beats summed pairwise CLIP losses by margins that widen with modality count.Source: C. Adding Minimality: IB-Regularized… · pp. 4–5
• The surveyed work frames CLIP’s objective as a special case of a broader IB-regularized family that can improve alignment in task- and architecture-specific settings.Examples include MVEB, which recasts multi-view IB as alignment plus differential entropy, and DVSIB, which unifies VAE, CLIP, and Barlow Twins objectives under stated assumptions.Source: C. Adding Minimality: IB-Regularized… · p. 5

D. Beyond Redundancy: PID-Informed Synergistic Contrastive Learning

• PID-informed contrastive learning challenges the assumption that task-relevant information is redundantly present in both modalities, decomposing it into redundant, unique, and synergistic components.Liang et al. show contrastive learning degrades as modality-unique task-relevant information grows.Source: D. Beyond Redundancy: PID-Informed… · p. 5
• 77.0% versus 71.4%: InfMasking raises accuracy on a controlled synergy benchmark by stochastically masking large proportions of each modality before fusion.CoMM theoretically preserves total task-relevant information under assumptions including minimal label-preserving augmentations, sufficient encoder expressivity, and optimization to the MI optimum, but does not identify the four atoms separately.Source: D. Beyond Redundancy: PID-Informed… · p. 5
• PID analysis finds Barlow Twins’ whitening losses destroy synergistic information along with redundant information; a recalibration protocol recovers it.DeCUR instead partitions Barlow Twins’ embedding space into common and unique subspaces, with validation on SAR-optical and RGB-depth data.Source: D. Beyond Redundancy: PID-Informed… · p. 5

III. COMPRESSING FOR ROBUST FUSION

• Naive MI-maximizing fusion can become brittle when modalities are missing, corrupted, or imbalanced, or when annotation artifacts create shortcuts.The section groups robust-fusion work by the sub-problem each compression mechanism targets and notes that MI- and IB-labelled methods can use near-identical machinery.Source: §III · p. 5

A. Foundational MI–IB Hybrid Fusion Architectures

• MVIB maximizes a lower bound on MI between two views while penalizing symmetrized KL divergence to discard view-private information; MIB makes the compression–relevance trade-off explicit.In sentiment and emotion recognition, MIB’s explicit compression beats uncompressed tensor or graph fusion; other methods use InfoNCE or KL-based formulations for multimodal fusion.Source: A. Foundational MI–IB Hybrid Fusion… · p. 5

B. Correcting Modality Imbalance and Distribution Shift

• OMIB sets per-modality weights analytically from conditional task-relevant information, while Elastic IB interpolates deterministic and stochastic IB objectives for distribution shift.OMIB’s proposed upper bound on β assumes idealized disjoint information sets; on MNIST-to-USPS transfer, intermediate Elastic IB mixing values outperform either pure objective.Source: B. Correcting Modality Imbalance and… · p. 6

C. Robustness to Missing Modalities

• CyIN uses one unified model to reconstruct a missing modality’s bottleneck latent from the remaining purified latents; RedCore adapts supervision to each modality’s missing rate.I3-MRec combines invariant risk minimization with missing-aware IB fusion and degrades by ∼12% in Recall@20 at 90% missing.Source: C. Robustness to Missing Modalities · p. 6

D. Robustness to Noise, Shortcuts, and Unreliable Modalities

• MCIB compresses each primary modality to retain information complementary given an auxiliary modality, addressing shortcuts in sarcasm detection; CaMIB targets shortcut information as an unobserved confounder.CaMIB combines per-modality IB filtering with an instrumental variable and backdoor adjustment, improving out-of-distribution accuracy.Source: D. Robustness to Noise, Shortcuts, and… · p. 6

E. Compression as Cross-Modal Knowledge Distillation

• AMID addresses teacher–student collapse by adding a second MI term to an auxiliary-modality model and an adversarial entropy bound.Xie et al. [65] prove a sufficient condition for distillation benefit in a jointly Gaussian, small-λ asymptotic model: teacher–student MI must exceed student–label MI; the converse is observed only empirically.Source: E. Compression as Cross-Modal… · pp. 6–7

F. Compression for Efficient Transmission, Extraction, and Retrieval-Adjacent Tasks

• In wireless semantic communication, Distributed IB prunes low-relevance modality–task links, cutting active links by up to 70% and communication rate by over 40% while matching full-participation baselines.Other examples recast an IB sufficiency objective as KL divergence to avoid MI estimation, apply per-modality variational IB to entity alignment, and report that lower camera–LiDAR MI predicts higher 3D-detection accuracy in experiments.Source: F. Compression for Efficient… · p. 7

IV. DECOMPOSING AND TYPING INTERACTION

• PID splits joint predictive information into redundant, unique, and synergistic terms, giving “private” information an operational vocabulary beyond simply “not shared.”PID-based methods extend beyond sentiment benchmarks into medicine, recommendation, and federated learning.Source: §IV · p. 7

A Pre-PID Approach

• Pre-PID multi-view clustering methods use MI maximization to align shared representations and IB-style compression to strip away information that is not shared.SIB-MSC separates superfluous from signal-relevant input MI, while DVIB uses four MI objectives for shared and private latents; neither guarantees that shared information is redundant or synergistic.Source: A Pre-PID Approach · p. 7

B. Foundational PID Theory and Estimators

• Williams and Beer [10]’s PID decomposes target–source MI into nonnegative redundant, unique, and synergistic pieces, with the atoms depending on the chosen redundancy measure.For a shared fair bit, redundancy is 1; for an XOR target of independent fair inputs, synergy is 1 and the other atoms are zero.Source: B. Foundational PID Theory and… · pp. 7–8
• PID estimators extend beyond discrete or Gaussian settings through copula-parametrized variational bounds, jointly Gaussian optimization, and MI-preserving normalizing-flow encoders.With only labeled unimodal and unlabeled multimodal co-occurrence data, synergy is generally unrecoverable; provable bounds can predict trained-model performance without model training.Source: B. Foundational PID Theory and… · p. 8

C. Frameworks for Quantifying Multimodal Interactions

• 95–100% of best-model performance: Liang et al. report this range for PID-based model selection across five synthetic and six real MultiBench datasets without training candidate models.Their CVX and BATCH estimators use discretized features and differentiable Sinkhorn–Knopp projection, respectively.Source: C. Frameworks for Quantifying… · p. 8
• 5% versus 1.8%: under missing-at-random shifts, naive PID analysis nearly threefold overstates a chest X-ray’s unique diagnostic information.ICYM2I applies inverse-probability weighting to training, evaluation, and PID estimation; InterSHAP generalizes interaction explanations to any number of modalities and reports more synergy than PID on a synthetic pure-synergy dataset.Source: C. Frameworks for Quantifying… · p. 8
• SynGR targets a synergy-linked proxy that also reflects redundancy and inter-modal dependence, gaining roughly 10% on average over alignment-centric recommenders.The proxy is I(Xv, Xt; Y) − [I(Xv; Y) + I(Xt; Y) − I(Xv; Xt)], which equals I(Xv; Xt | Y) and is not the synergy atom itself.Source: C. Frameworks for Quantifying… · p. 8

D. PID-Guided Representation Disentanglement

• PID-guided disentanglement extends shared/private representations with redundancy, uniqueness, and synergy, while Table IV distinguishes targeted atoms from estimators that may use proxies rather than exact PID.The methods integrate interaction typing into representation-learning objectives.Source: D. PID-Guided Representation… · p. 8, Table IV · p. 9
• MISA established invariant and private subspaces using distributional-distance surrogates, becoming a canonical baseline despite not formally decomposing redundancy, uniqueness, and synergy.Its invariant space uses Central Moment Discrepancy alignment, while soft orthogonality keeps the private space non-redundant.Source: D. PID-Guided Representation… · p. 8
• PID-guided sentiment and regression methods add hierarchical or bounded-optimality objectives, with reported redundancy dominance in CT-slice regression and synergy dominance in vision–text sentiment.FINE beats MISA and later contrastive baselines on four sentiment datasets; DisentangledSSL is exactly optimal when the Minimum Necessary Information point is attainable.Source: D. PID-Guided Representation… · p. 8
• Clinical and other applications use PID-typed factorization for weakly supervised survival prediction, missing-modality fusion, noise filtering, label scarcity, and communication across federated agents.PIBD achieves the best concordance index across five TCGA datasets, while PARSE exchanges redundant slices and shares synergy-head parameters among agents with the same modality set.Source: D. PID-Guided Representation… · p. 9

E. Interaction-Aware Mixture-of-Experts Architectures

• A direct architectural payoff is one expert per interaction type, with sample routing replacing post-hoc measurement of redundancy, uniqueness, and synergy.MMOE uses a prediction-discrepancy heuristic; I2MoE learns expert specialization end-to-end, and PathMoE adds a cell-graph modality and fifth expert.Source: E. Interaction-Aware… · p. 9

V. INTERPRETABILITY, EFFICIENCY, AND DIAGNOSIS IN THE FOUNDATION-MODEL ERA

• The foundation-model section shifts MI, IB, and PID from shaping representations to pruning, tokenizing, routing, attributing, and diagnosing pretrained models.Methods include post-hoc probes on frozen models and interventions during lightweight adaptation, routing, tokenization, or data selection.Source: §V · p. 10

A. Efficiency: Pruning, Tokenization, and Routing at Scale

• Uniform compute allocation motivates efficiency methods; MI-Pruner retains 5.57% of Video-LLaVA tokens at 93.19% relative accuracy.It frames pruning as pointwise mutual information maximization and has a (1 −1/e) guarantee under the paper’s conditional-independence assumption.Source: A. Efficiency: Pruning, Tokenization,… · p. 10
• InfoTok applies an Information Bottleneck trade-off to a shared visual tokenizer, reducing redundant image information while preserving sufficiency and text alignment.It improves understanding and generation benchmarks simultaneously with no extra data.Source: A. Efficiency: Pruning, Tokenization,… · p. 10
• MERGE routes temporally lagged sensor interactions using redundancy, uniqueness, and synergy, while MINT groups instruction-tuning datasets by interaction scores and beats unselective tuning by double-digit margins on HEMM.MERGE steers an RUS-aware mixture-of-experts router; MINT fine-tunes interaction-coherent dataset clusters separately.Source: A. Efficiency: Pruning, Tokenization,… · p. 10

B. Interpretability and Attribution for Pretrained Multimodal Models

• IB attribution measures information flow through a bottleneck, whereas PID attribution distinguishes redundant, modality-unique, and synergistic information.M2IB uses the paired modality embedding as a proxy target for label-free CLIP-style attribution; NIB removes stochastic noise and sensitive β, with importance equal to mutual information lost.Source: B. Interpretability and Attribution… · p. 10
• DiffusionPID uses prompt-phrase and generated-pixel PID maps to expose occupational biases and localize homonym disambiguation; its redundancy-guided interventions remove words where MI- and attention-based maps cannot.A training-free VLM attribution study also finds fusion direction systematically biases which modality appears dominant.Source: B. Interpretability and Attribution… · p. 10
• Layer-wise analysis finds LLaVA’s modal transduction ends at 82% language-unique, 6% vision-unique, and under 2% synergy; across 26 LVLMs, synergy emerges during visual instruction tuning.The LLaVA finding was validated causally via attention knockout.Source: B. Interpretability and Attribution… · p. 10

C. Diagnosing Failure Modes: Modality Collapse and the Modality Gap

• Across foundation models, non-text information can survive encoding despite decoder failures, while targeted LoRA fixes some failures; IB methods also address visual forgetting and shift robustness.The reviewed work includes a GMI-Wasserstein bound that can be loose at high mismatch, and a Vittle evaluation across 45 datasets and 30 shift scenarios.Source: C. Diagnosing Failure Modes: Modality… · pp. 10–11

VI. SYNTHESIS: A UNIFYING COORDINATE SYSTEM

• The Bottleneck PID Identity expresses I(Z1, Z2; Y) as four nonnegative atoms, showing that an IB sufficiency term sums qualitatively distinct information types.The identity applies to the encoder pair before fusion; a fused representation retains it only up to information discarded by fusion.Source: §VI · pp. 11–12
• The GMIL organizes surveyed objectives by compression costs and typed PID rewards, mapping untyped MI, multimodal IB, synergy-maximizing contrastive learning, and uniqueness-favoring disentanglement to four corners.These mappings are exact or approximate; label-free alignment uses a target substitution, and multimodal IB often compresses a fused representation.Source: §VI · p. 12
• GMIL synthesizes existing practice rather than proposing a new algorithm, and its typed rewards distinguish synergy-favoring from redundancy-favoring objectives that scalar joint-information rewards cannot distinguish.MRdIB [134] is an approximate GMIL instance that predates the survey’s formulation.Source: §VI · p. 12
• The screened corpus leaves several GMIL parameter regions unoccupied, including redundancy-only and synergy-suppressing rewards; these are census findings, not proof that no such methods exist.Candidate objectives are practically realizable only when their rewarded atoms admit tractable, stable estimators or surrogates.Source: §VI · p. 12
• The coordinate system offers a heuristic design guide: a task’s measured dominant atom and deployment constraint suggest candidate regimes to test, not optimality guarantees.The choice of a method’s target interaction profile is an empirical question addressed before training.Source: §VI · p. 13
• With exogenous encoder noise, compression cannot increase the aggregate sum of the four PID atoms beyond raw joint task information.Per-modality Markov chains alone do not establish the joint bound; correlated encoder noise can violate the required joint factorization.Source: §VI · p. 13
• The aggregate compression bound holds across redundancy measures, but term-by-term claims depend on measure-specific monotonicity, which surveyed PID estimators do not guarantee uniformly.The synthesis links MI, IB, and PID as progressively refined views of dependency, compression, and interaction types.Source: §VI · pp. 11, 13

• Comparing MI-, IB-, and PID-based treatments of the same problem makes visible where a framework has not yet been tried.Source: §VII · p. 14
• The section collects three gaps in the surveyed literature.Source: §VII · p. 14
• The first gap spans both information estimation and evaluation across frameworks.Source: §VII · p. 14

• No general-purpose, sample-efficient PID estimator exists for the unrestricted continuous, high-dimensional case.The surveyed methods use specialized fixes, while PID requires estimating four non-negative terms.Source: A. Estimator Intractability at Scale,… · p. 14
• A distribution-free high-confidence lower bound on mutual information from N samples is capped at O(ln N), and PID estimators rely on specialized restrictions or methods.The examples range from copula-based bounds and jointly Gaussian restrictions to Thin-PID and Flow-PID using normalizing flows.Source: A. Estimator Intractability at Scale,… · p. 14
• CVX and BATCH scale poorly with category count, while correctly estimated PID can remain biased under realistic missing-at-random shifts between training and deployment.Pointwise LSMI addresses the category-count gap; PID’s four simultaneously estimated non-negative terms compound the estimation difficulty.Source: A. Estimator Intractability at Scale,… · p. 14
• MI-, IB-, and PID-based treatments of identical problems are compared narratively rather than on a common benchmark, motivating a suite that fixes task, data, and architecture while varying the objective.Source: A. Estimator Intractability at Scale,… · p. 14

B. Scaling PID Beyond Two or Three Modalities

• General, scalable PID estimation for four or more genuinely distinct modalities remains largely unsolved.Most tractable estimators are derived and benchmarked for two sources and one target; recent multi-source or tri-modal extensions are narrow, and some methods provide bounds rather than exact estimates.Source: B. Scaling PID Beyond Two or Three… · p. 14

C. From Post-Hoc Diagnosis to Training-Time Integration in Foundation Models

• The surveyed corpus has no PID-based diagnosis or fix for foundation-model modality collapse or modality gaps, and no PID regularizer in pretraining objectives.Existing PID analyses characterize normally behaving models, while PID-guided grouping and routing operate at data-selection or routing levels.Source: C. From Post-Hoc Diagnosis to… · p. 14
• For continuous modalities, PID atoms depend on the redundancy measure and estimator family; observational PID also cannot distinguish causal unique information from a dataset shortcut.The stability of PID-guided training across these choices is uncharacterized, and privacy and fairness implications remain unexplored.Source: C. From Post-Hoc Diagnosis to… · p. 14

VIII. CONCLUSION

• MI, IB, and PID refine one inquiry into inter-modal dependency: how much modalities depend on each other, what task-relevant information survives compression, and what kind of dependency is preserved.The survey organizes 170 recent works and 12 foundational references within this framework.Source: §VIII · pp. 14–15
• The Generalized Multimodal Information Lagrangian and related identities place surveyed methods in a shared coordinate system whose unoccupied regions point toward model families not yet built.Source: §VIII · p. 15
• MI, IB, and PID are increasingly applied to prune, attribute, and diagnose pretrained models, while integrating redundancy, uniqueness, and synergy into pretraining remains a future direction.Source: §VIII · p. 15