PaperlayerView exact revision on arXiv

Source-linked AI summary

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Songyuan Sui, Zhen Tan, Mohan Zhang, Rana Muhammad Shahroz Khan, Xia Hu, Tianlong Chen

arXiv:2610.04792v2cs.AIcs.CV

TL;DR

The paper examines multimodal models when a modality is unavailable at inference. It characterizes associated subspace rotations and proposes Geodesic Unlearning; experiments show improved missing-modality performance while largely preserving full-modality accuracy.

  • Problem

    At deployment, modalities may be unavailable, and the performance advantage of joint multimodal training may not transfer to that setting.

    Source: §1 · p. 1

  • Method

    Geodesic Unlearning realigns a selected principal subspace toward a unimodal reference along a Grassmannian geodesic, with optimal alignment under a fixed subspace-distance budget.

    Source: §7 · p. 10

  • Results

    Across fusion, CLIP-style two-tower, and vision-language models, GU improves missing-modality performance while largely preserving full-modality accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support a geometric view of missing-modality degradation and localized subspace editing as a route to robustness correction.

    Source: §1 · p. 2, §7 · p. 10

    GU’s gains are smaller in CLIP-style and BLIP architectures, where relevant dependencies may be more distributed than in fusion models.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.

1 Introduction

The paper identifies a deployment-time failure in which multimodal models can underperform unimodal models when an input modality is absent, and links this degradation to learned cross-modal dependencies. It proposes Geodesic Unlearning, a post-training subspace-editing method intended to improve missing-modality robustness while preserving full-modality utility.

1 Introduction

  • With one modality absent at inference, full-modality-trained models can underperform models trained only on the available modality across diverse architectures.The paper reports a recurring ordering: full-modality inference performs best, unimodal training next, and full-modality training with a missing input worst.Source: §1 · pp. 1–2
  • Missing-modality degradation is associated with principal-subspace rotations in cross-modal interaction layers and greater task-aware representation harm on misclassified samples.The representation-harm gap persists toward the classifier.Source: §1 · p. 2
  • Geodesic Unlearning edits a selected layer by rotating its principal input subspace toward a matched unimodal reference along a Grassmannian geodesic.The method retains the layer’s principal coefficients, output factor, and residual; its correction is geometrically optimal within a fixed subspace-distance budget.Source: §1 · p. 2
  • Across fusion, CLIP-style two-tower, and vision-language models, GU improves missing-modality performance while largely preserving full-modality accuracy.Comparisons include multimodal-specific methods, general training and adaptation baselines, and parameter-editing controls.Source: §1 · p. 2
  • The paper frames the contribution as characterizing the failure mode, relating it to parameter- and representation-level geometry, and correcting it through localized subspace editing.The proposed method is lightweight and theoretically grounded.Source: §1 · p. 2

2 Related work

Prior work addresses missing modalities through recovery, adaptation, or modality balancing, while this paper focuses on harmful parameter dependencies retained when a modality is absent at deployment. It connects subspace geometry and machine unlearning to a model-side correction of those dependencies.

2 Related work

  • Missing-modality methods include data- and representation-level recovery, adaptive architectures or strategies, and training-time modality balancing.This paper instead studies dependencies retained in an already-trained model when a modality is completely absent at deployment.Source: §2 · p. 2
  • Prior geometric work studies structured representations, low-dimensional parameter subspaces, and Grassmannian geodesic updates; this paper connects those perspectives to missing-modality robustness.Its account links principal-subspace rotations with task-aware representation harm under missing inputs.Source: §2 · pp. 2–3
  • The paper repurposes machine unlearning to selectively attenuate harmful cross-modal dependencies while preserving useful multimodal capability.It presents geometric parameter editing as a way to address missing-modality robustness in an already-trained model.Source: §2 · p. 3

3 Problem setup and preliminaries

The paper studies models trained with full modalities and evaluated when one modality is unavailable, focusing on a single missing modality in bimodal text-image systems. It defines evaluation gaps and compares modality-specific parameter subspaces against a matched unimodal reference.

3.1 Deployment-time missing modality setting

  • The deployment setting evaluates models trained on full-modality inputs when one or more modalities are unavailable, with the study focused on a single missing modality.For bimodal systems, the same modality is absent for every test example while the other remains available; a null input can be implemented with a masking token, zeroed stream, or learned embedding.Source: §3.1 · p. 3

3.2 Evaluation protocol and missing-modality gap

  • For text-image models, TI-TI denotes full-modality training and testing, T-T text-only training and testing, and TI-T full-modality training followed by text-only testing.The missing-modality gap measures performance loss from removing the image at deployment; the unimodal robustness gap measures how much TI-T underperforms T-T.Source: §3.2 · p. 3

3.3 Parameter subspace geometry

  • The analysis covers fusion, aligned two-tower, and vision-language model families to represent distinct multimodal integration mechanisms.It compares principal input and output subspaces of multimodal and unimodal layers using singular vectors and subspace metrics such as principal angles, overlap, and Frobenius distances.Source: Table 1 · p. 4, §3.3 · p. 4

4 Empirical analysis of missing-modality degradation

Across fusion, two-tower, and vision-language architectures, missing-modality inference generally underperforms unimodal training. Its degradation is associated with principal-subspace rotations and task-aware representation harm concentrated around cross-modal interaction stages.

4.1 Analysis scope: architectures and benchmarks

  • The analysis covers fusion, two-tower dual-encoder, and vision-language model families under deployment-time missing-image evaluation.The study examines whether degradation is architecture-specific or broader across these families.Source: §4.1 · p. 4

4.2 RQ1: Is missing-modality degradation consistent across architectures?

  • Across 27 architecture–dataset settings, full-modality inference outperforms unimodal training, which outperforms missing-modality inference in 26 settings; unimodal training beats missing-modality inference in all 27.The missing-modality gap averages 4.74 F1 points and ranges from 0.08 to 20.29.Source: §4.2 · p. 4

4.3 RQ2: Does parameter geometry track degradation?

  • Across 27 settings, mean principal angle and missing-modality gap have a strong positive association (r = 0.91, R2 = 0.82) after normalization, with positive associations also present in raw measurements.The association is reported for fusion, stronger-encoder fusion, and CLIP-style two-tower models; the three-point BLIP-2 trend is descriptive only.Source: Figure 2 · p. 5, §4.3 · p. 5

4.4 RQ3: Where does representation harm concentrate?

  • Normalized stage-wise harm gaps are already evident at cross-modal interaction stages and often persist near the classifier, indicating downstream persistence rather than a classifier-only effect.Final-error samples exhibit larger harm near the classifier, and interaction stages account for a substantial share of the gap across settings.Source: §4.4 · pp. 5–6, Figure 3 · p. 6

5 Geodesic Unlearning

Geodesic Unlearning edits one selected layer of a multimodal model toward a matched unimodal reference. It uses an angle–harm criterion for selection and a Grassmannian geodesic to realign the principal input subspace while preserving other layer components.

5.1 Overview

  • GU applies a lightweight post-training correction to one selected layer, rotating its principal input-side subspace toward a matched unimodal reference while retaining useful multimodal functionality.The procedure applies across fusion, CLIP-style two-tower, and BLIP-2 models, with the target layer varying by architecture.Source: §5.1 · p. 6

5.2 Target-layer selection

  • GU selects the layer with the highest development-set score formed by adding within-model z-scores of principal angle and the missing-modality harm gap.The test set is reserved for final evaluation.Source: §5.2 · pp. 6–7

5.3 Subspace realignment and reconstruction

  • GU moves the principal input subspace along a Grassmannian geodesic toward the unimodal reference, with η = 0 leaving it unchanged and η = 1 reaching the reference endpoint.Reconstruction retains the multimodal output factor, principal coefficients, and residual structure; the geodesic edit is as close to the reference as any edit within the same subspace-distance budget.Source: §5.3 · p. 7

6 Experiments

The experiments evaluate GU across three missing-image benchmarks, model families, and baseline groups, then assess robustness–preservation, cost, and ablation behavior. GU leads the compared methods on all three datasets, while its gains depend on architecture and target-layer choice.

6.1 Experimental setup

  • Benchmarks comprise binary-classification Hateful Memes, SNLI-VE, and IU-XRay, evaluated primarily with F1; A-OKVQA is an additional multiple-choice benchmark.The main comparison uses a RoBERTa–ResNet50 cross-attention fusion setting, with broader robustness and preservation evaluation across the model families in Table 1.Source: §6.1 · pp. 7–8, Figure 4 · p. 8

6.2 Main results in the deployment-time missing-modality setting

  • +2.38 F1 on Hateful Memes, +1.27 on IU-XRay, and +2.21 on SNLI-VE over each dataset’s strongest baseline; GU ranks first on all three.These are three-run mean F1 results for complete image absence at inference, achieved by editing one selected layer after training.Source: §6.2 · p. 8, Table 2 · p. 8
  • With one modality entirely unavailable, recovery and adaptive-fusion baselines are less consistent; GU instead edits model parameters and further improves the competitive null-token pathway.SMIL and Flex-MoE are competitive on SNLI-VE but do not consistently improve over the null-token baseline across datasets, while DyMo degrades substantially.Source: §6.2 · p. 8
  • On SNLI-VE, GU and layer interpolation preserve similar broad representation layouts, while quantitative analysis finds larger error-margin gains and more corrected errors with GU.The latter comes with a small increase in regressions, while both edits retain high similarity to the original TI-T representations.Source: Figure 4 · p. 8, §6.2 · p. 9

6.3 Robustness–preservation trade-off across model families

  • GU improves TI-T across model families while keeping TI-TI nearly unchanged, and usually gives larger TI-T gains and smaller TI-TI degradation than layer interpolation.Gains are strongest for fusion models, where cross-modal dependencies are more localized; they are smaller for CLIP-style and BLIP architectures with more distributed dependencies.Source: §6.3 · p. 9, Table 3 · p. 9

6.4 Cost analysis

  • Including reference training, GU uses 7–22% less wall-clock time and 16–62% less peak GPU memory than Dual CE across three architectures.GU’s one-time post-training edit takes 230–858 seconds, reduces time by 57–88% relative to post-hoc LoRA, and adds no parameters or forward-pass FLOPs.Source: §6.4 · p. 9, Table 4 · p. 10

6.5 Ablation studies

  • GU is relatively stable across tested geodesic coefficients and ranks, while target-layer selection has a more decisive effect on recovery.Editing the selected cross-modal interaction layer attn_v_from_t_in produces the largest TI-T improvement while maintaining strong TI-TI performance; editing most other layers yields limited gains.Source: Figure 4 · p. 8, §6.5 · p. 10

7 Conclusion

The conclusion frames missing-modality degradation as a deployment-time failure associated with cross-modal subspace rotations and representational harm. It presents GU as a theoretically grounded localized correction with a favorable robustness–preservation trade-off.

7 Conclusion

  • The paper concludes that GU supports a geometric account of missing-modality degradation and offers localized subspace editing as a practical correction strategy.The conclusion also reports supporting evaluation on multiple-choice VQA and missing-text settings.Source: §7 · p. 10
  • GU realigns a selected principal subspace toward a unimodal reference and is provably optimal under a fixed subspace-distance budget.The method treats the subspace as a point on the Grassmannian.Source: §7 · p. 10
  • The characterized failure occurs when full-modality models can underperform unimodal models with a modality absent at inference, and is associated with subspace rotations and task-aware representation harm.Source: §7 · p. 10

A Limitations

The paper bounds its evaluation and identifies broader deployment risks, while supplementary results reinforce the missing-modality performance ordering and localize parameter-space changes to interaction layers.

A Limitations

  • Evaluation covers classification, including multiple-choice visual QA, but not free-form multimodal generation or large-scale experiments on substantially larger models.The authors also report targeted preprocessing and complementary controls for artifact concerns on Hateful Memes and SNLI-VE, with further evaluations reproducing the findings without those artifacts.Source: A Limitations · p. 18

B Broader impacts

  • Greater robustness may encourage deployment despite possible incorrect predictions, and it does not remove biases in training data.The authors identify incomplete inputs from sensor failure, privacy constraints, or deployment limitations as relevant real-world settings.Source: B Broader impacts · p. 18

C.1 More results on missing-modality degradation

  • The ordering TI-TI > T-T > TI-T holds in 26 of 27 settings, while T-T > TI-T holds in all 27.The sole non-strict case is strong-encoder Fusion Cross-Attention on Hateful Memes, where TI-TI and T-T both score 66.93 F1, above TI-T at 66.85 F1.Source: §C.1 · p. 18

C.2 More results on parameter subspace rotation

  • Projection and cross-modal interaction layers generally have larger principal-angle deviations and lower subspace overlap than classifier layers.The final classifier has near-perfect overlap and a near-zero output-side angle; interaction-layer geometry is proposed as a diagnostic and target-selection criterion for GU.Source: §C.2 · p. 18

C.3 More results on representational harm under missing modalities

Layer-wise results show that task-aware representational harm concentrates around multimodal interaction stages and often persists near the classifier, complementing the parameter-subspace analysis.

C.3 More results on representational harm under missing modalities

  • Cross-modal interaction layers show both substantial subspace rotation and nontrivial representational harm, whereas image projection layers have zero harm under missing-image evaluation.Image projection layers can still exhibit large subspace rotations; the image modality is unavailable in this evaluation.Source: §C.3 · p. 19, Table 7 · p. 20, Table 6 · p. 20
  • Representational harm becomes pronounced around multimodal interaction stages and often persists into the classifier input.The layer-wise table reports error–correct harm and violation-rate gaps; larger gaps indicate that error samples occupy more task-inconsistent regions than correct samples.Source: §C.3 · pp. 18–19
  • The layer-wise findings support interpreting interaction-layer geometry as accompanying task-relevant representation shifts under missing-modality inputs.Source: §C.3 · p. 19

D Methodology details

The method constructs a Grassmannian geodesic that rotates a selected multimodal input subspace toward its unimodal reference. Its edit-budget guarantee is geometric, and implementation reconstructs only the selected layer.

D.1 Grassmannian geodesic construction

  • GU pairs principal directions of the multimodal and matched unimodal top-k input subspaces, then rotates the multimodal basis toward the reference along a geodesic.The path starts at the original multimodal basis and ends in the reference subspace, expressed in the multimodal model’s coordinates.Source: §D.1 · p. 21

D.2 Geometric rationale and path comparison

  • GU retains the selected layer’s output factor, principal coefficients, and residual while editing the orientation of its dominant input directions.The layer’s top-k right singular subspace represents the dominant input directions it reads.Source: §D.2 · p. 21
  • For edit strength η, the geodesic advances each principal direction through ηθ_i, leaving (1−η)θ_i to the reference.The subspace-distance budget is the allowed departure from the original multimodal subspace.Source: §D.2 · p. 21
  • Within a fixed subspace-distance budget, the geodesic point gets as close as possible to the reference; this optimality does not assert optimal task performance.Source: §D.2 · p. 21
  • Unlike GU’s proportional angular progress, aligned linear interpolation generally does not move every principal direction through the same fraction of its angle.For the linear path, the angular displacement is ϕ_i(η), which generally differs from ηθ_i.Source: §D.2 · p. 22

D.3 Implementation notes

  • The implementation reconstructs the selected layer using the edited basis and retained factors, leaving all other layers unchanged.The η = 0 conversion recovers the original multimodal layer.Source: §D.3 · p. 22

E.1 Dataset

This appendix evaluates GU’s missing-modality recovery and full-modality preservation on binary-benchmark and multiple-choice VQA settings, and examines geometric editing choices. The reported results show improved missing-modality performance, with full-modality performance generally preserved but a measured trade-off on VQA.

E.1 Dataset

  • The experiments use Hateful Memes [26], SNLI-VE [59], and IU-XRay as binary classification benchmarks, selecting hyperparameters on development sets and reporting held-out test results.For SNLI-VE, neutral examples are removed from the original three-class task. Evaluations include matched-backbone missing-modality methods and architecture-compatible alternatives; GU uses a single-layer post-training edit, with η = 0.75 and k = 8 selected on development data and fixed across datasets and model families.Source: §E.1 · p. 22, §E.2 · p. 22, §E.2.1 · p. 23

F.1.1 Three-run means and standard deviations

  • GU beat the strongest competing method in all 30 paired runs, with mean F1 gains of +2.41 on HM, +1.34 on IU-XRay, and +2.21 on SNLI-VE; all six comparisons remained significant after Holm correction (p < 0.05).Three-run results also retain GU’s highest mean TI-T F1 on all three datasets, while family-level results show it matches or improves interpolation’s recovery and preservation.Source: §F.1.2 · p. 26, §F.1.1 · p. 26, Table 17 · p. 27

F.2 More details on representation-level evidence

  • For SNLI-VE Fusion Cross-Attention, GU’s error-margin gain was +0.30 versus +0.06 for interpolation, correcting 34.0% versus 11.8% of original errors, with regression rates of 5.1% versus 4.5%.Both edits retained high representation similarity (linear CKA 0.991 for GU and 0.988 for interpolation); GU’s TI-T F1 was 74.41 versus 67.29 for interpolation.Source: Table 18 · p. 27, §F.2 · p. 27

F.3 More results on robustness–preservation trade-off

  • Across SNLI-VE variants, GU generally produced larger TI-T gains than interpolation with smaller or comparable TI-TI changes; gains were strongest in cross-attention fusion and smaller for CLIP-style and BLIP-2 models.The pattern is consistent with more distributed cross-modal interactions in CLIP-style and BLIP-2 architectures.Source: §F.3 · p. 27, Table 19 · p. 28

F.4.1 MoRA and OGM-GE

  • Across nine settings, GU achieved the highest mean TI-T in every setting and improved TI-T by 0.94 F1 on average while changing TI-TI by only −0.41 F1.Across six CLIP comparisons, its average advantages over MoRA [73] and OGM-GE [37] were 4.42 and 5.04 F1; the largest TI-TI loss was 1.08 F1 on IU-XRay Fusion Concat.Source: §F.4.1 · p. 28, Table 20 · p. 29

F.4.2 LoRA and Dual CE

  • Against LoRA, GU averaged +3.17 F1 TI-T improvement and −0.36 F1 TI-TI change, versus LoRA’s +2.44 and −1.99 F1, respectively.GU outperformed LoRA on both metrics for Fusion CA and CLIP MP; on BLIP-2, LoRA recovered slightly more TI-T but reduced TI-TI, while GU left both essentially unchanged. Dual CE improved TI-T over Null-token by 13.57, 2.16, and 4.41 F1 on Fusion CA, CLIP MP, and BLIP-2, respectively, while GU preserved higher full-modality performance on all three; applying GU after Dual CE helped on Fusion CA but not CLIP or BLIP-2.Source: §F.4.2 · pp. 28–30, Table 21 · p. 29

F.5.1 Target-layer selection

  • Joint angle–harm target selection yielded the largest TI-T gain (+1.41 F1) and smallest TI-TI loss (−0.13 F1), outperforming angle-only and harm-only selection.Angle-only produced +0.34 TI-T and −0.61 TI-TI; harm-only produced +0.72 TI-T and −0.36 TI-TI.Source: Table 22 · p. 30, §F.5.1 · p. 30

F.5.2 Editing strategy comparison

  • On SNLI-VE, GU reached 74.41 TI-T F1 versus 67.29 for Weight and 66.84 for Linear, while maintaining comparable TI-TI performance.GU had the highest mean TI-T and TI-TI in all three dataset settings; comparison with Linear isolates the benefit of the geodesic path when target subspace, rank, coefficient, and retained components are matched.Source: Table 23 · p. 30, §F.5.2 · pp. 30–31
  • GU combines geometric deviation with task-aware harm to select targets, then corrects the selected principal input subspace along a Grassmannian geodesic.Source: §F.5.2 · p. 31

F.6 Generalization to multiple-choice VQA

  • The evaluation extends GU to A-OKVQA multiple-choice VQA using Qwen2.5-3B and OpenCLIP ViT-bigG-14 encoders with official dataset splits.Accuracy is reported as mean ± sample standard deviation over ten independent runs.Source: §F.6 · p. 31
  • On A-OKVQA, TI-TI > T-T > TI-T accuracy (50.79 > 47.17 > 38.94), while shuffling images lowers full-input accuracy from 50.79% to 39.85%.TI-T and TI-TI denote missing-image and full-modality evaluation; the T reference reports T-T accuracy in the TI-T column.Source: §F.6 · p. 31
  • GU improves TI-T accuracy by 2.13 points and has the highest TI-TI accuracy among interventions, 3.39 points below the original TI model.GU’s TI-TI accuracy exceeds Interpolation and LoRA by 4.19 and 2.39 points; LoRA recovers more TI-T accuracy (+4.58) but incurs a larger TI-TI loss (−5.78).Source: §F.6 · p. 31

F.7 Complementary evaluation under missing-text inputs

Under missing-text inputs, GU improves TI-I across HM, IU-XRay, and SNLI-VE while changing full-modality TI-TI performance only slightly. GU also outperforms matched layer interpolation on TI-I and narrows, but does not eliminate, the gap to the image-only reference.

F.7 Complementary evaluation under missing-text inputs

  • The evaluation uses Fusion Cross-Attention on HM, IU-XRay, and SNLI-VE; TI-I is a TI-trained model evaluated without text, and I-I is the matched image-only reference.Both GU and layer interpolation use I-I as the reference; Table 25 reports F1 as mean ± sample standard deviation over three independent runs.Source: §F.7 · pp. 31–32
  • GU improves TI-I by 1.02, 0.84, and 2.13 F1 on HM, IU-XRay, and SNLI-VE, respectively, while the ordering TI-TI > I-I > TI-I holds on all three datasets.The ordering mirrors the missing-image pattern in Section 4.2.Source: §F.7 · p. 32
  • GU changes TI-TI by only −0.17, +0.07, and +0.03 F1 on HM, IU-XRay, and SNLI-VE, respectively.Source: §F.7 · p. 32
  • GU achieves higher mean TI-I than matched layer interpolation on all three datasets and remains closer to original TI-TI performance.Edited TI-I remains below the matched I-I reference, but GU consistently narrows that gap.Source: §F.7 · p. 32
Loading 2610.04792v2…