PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Perturbation Magnitude: Direction-Dependent Responses in Multimodal Geometric Representations

Yongsheng Luo, Wengan He, Yu Li, Rouying Wu, Wei Lv

arXiv:2610.08533v1cs.CVcs.LGcs.SD

TL;DR

The paper asks whether perturbation magnitude alone organizes changes in multimodal geometric scores. It tests controlled degradations and derives a first-order directional response from the Gramian-volume gradient. That directional quantity explains response variation strongly, while magnitude alone explains little.

  • Problem

    The paper asks whether scalar perturbation magnitude is sufficient to organize responses of multimodal geometric scores, rather than leaving substantial response variation unexplained.

    Source: §I · p. 1

  • Method

    The study applies controlled video blur and audio noise, measures displacement in pairwise-similarity geometry, and defines DGR as that displacement projected onto the local volume gradient.

    Source: §I · p. 1

  • Results

    83.8%–96.9% out-of-sample variance is explained by the directional first-order quantity across four dataset–degradation cells, while displacement magnitude explains at most 15% of out-of-sample variance in absolute volume response.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Geometric perturbation responses are condition-dependent: the same amount of relational movement can point in different directions relative to local geometry.

    Source: §IX · p. 11

    Evidence covers two datasets, two perturbation families at tested strengths, and one Gramian-volume geometry; DGR uses displacement observed after degradation.

    Source: D. Limitations and Scope · p. 10

Abstract

from arXiv · show

Geometric alignment scores based on Gram determinants provide a compact way to model higher-order consistency among modalities, yet how such scores respond to modality degradation is poorly understood. This paper asks whether the response of a multimodal geometric score is determined primarily by the magnitude of the perturbation-induced displacement. Using frozen cohorts from MSR-VTT (N=878) and DiDeMo (N=980), we apply controlled video blur and audio noise and analyze the response in the relational geometry on which the score is defined. Displacement magnitude explains at most 15% of the out-of-sample variance in the absolute response, and magnitude-matched pairs respond systematically differently, so scalar magnitude does not organize the response. The closed-form first-order expansion of the Gramian volume yields the Directional Geometric Response (DGR): the projection of the displacement onto the local volume gradient, which jointly captures the clean operating point, displacement magnitude, and displacement direction. The absolute first-order DGR term explains the observed response with out-of-sample R^2 of 0.838-0.969, matched-magnitude ranking accuracies of 0.864-0.963, and response-sign accuracies of 0.909-0.989, whereas the tested direction-free alternatives remain weak or unstable under the corresponding evaluation protocols. A pre-specified gain-normalization candidate, V/(g_V+eps), fails its predictability and clean-order gates. DGR uses the observed degraded-state displacement and is therefore an explanatory quantity, not a deployment-time predictor: geometric response depends on where the representation operates, how far degradation moves the relational geometry, and in which direction it moves.

I. INTRODUCTION

The introduction tests whether perturbation magnitude alone organizes changes in multimodal geometric scores. It reports substantial response heterogeneity and motivates a first-order directional account, Directional Geometric Response (DGR).

I. INTRODUCTION

  • The paper asks whether scalar displacement magnitude organizes geometric-score responses, extending the question beyond pairwise similarity to higher-order multimodal volume scores,.The motivation is that geometric scores are used in learning objectives and thus matter for degraded inputs.Source: §I · p. 1
  • Displacement magnitude explains at most 15% of out-of-sample variance in absolute volume response, and magnitude-matched pairs can respond very differently.The analysis uses controlled video blur and audio noise on MSR-VTT and DiDeMo.Source: §I · p. 1
  • DGR is the projection of relational displacement onto the local Gramian-volume gradient, combining the clean operating point, displacement magnitude, and displacement direction.The displacement is measured in the vector of pairwise cosine similarities among text, video, and audio embeddings.Source: §I · p. 1
  • 83.8%–96.9% out-of-sample variance, 0.864–0.963 matched-magnitude pairwise accuracy, and 0.909–0.989 response-sign accuracy are achieved by the directional first-order quantity across four dataset–degradation cells.Direction-free alternatives are weak or unstable under the corresponding evaluation protocols.Source: §I · p. 2
  • The pre-specified V/(gV + ε) gain normalization does not consistently improve response predictability and changes clean geometric ordering beyond the stated tolerance.The paper closes this tested normalization family rather than reformulating it after observing results.Source: §I · p. 2
  • The paper frames DGR as an empirical mechanism discovery, not a new mathematical theory, and treats it as explanatory rather than an inference-time predictor.The stated scope is a concrete geometric representation under controlled degradation, not a new robustness method or universal law.Source: §I · p. 2

II. RELATED WORK

Prior work develops shared multimodal spaces, joint geometric alignment scores, and methods for missing or unreliable modalities. This paper instead examines how controlled degradation changes a fixed geometric score, focusing on the direction of displacement in relational geometry.

A. Multimodal Representation and Geometric Alignment

  • GRAM measures joint Gramian volume and HyperGRAM extends it with Euclidean–hyperbolic geometry, while prior studies mainly evaluate representation quality and retrieval.Here the representation is held fixed and the geometric score's response to controlled modality perturbation is examined.Source: A. Multimodal Representation and… · p. 2

B. Multimodal Missingness, Perturbation, and Reliability

  • Missingness and reliability methods motivate a scalar-quality hypothesis, but modality quality need not specify the relational direction of the resulting displacement.The paper tests an already-trained geometric score rather than optimizing a model to compensate for missing or unreliable modalities.Source: B. Multimodal Missingness,… · p. 3

C. Local Perturbation and Representation Analysis

  • Gradient-based sensitivity methods study predictions or perturbations, whereas this analysis tests measured Gramian-volume changes using the closed-form derivative and observed relational displacement.The paper distinguishes its controlled signal degradations on a frozen encoder from adversarial-robustness analysis.Source: C. Local Perturbation and… · p. 3
  • Existing multimodal work mainly learns, aligns, completes, or robustifies representations, leaving the response of geometric scores to displacement direction underexplored.The paper distinguishes this question from sensitivity to perturbation strength and studies the score's local response in its own relational geometry.Source: C. Local Perturbation and… · p. 3

III. GEOMETRIC FORMULATION AND EXPERIMENTAL PROTOCOL

The formulation represents text, video, and audio through normalized embeddings and their pairwise cosine similarities, then measures degradation as a change in the resulting relational coordinates and geometric volume.

A. Multimodal Relational Geometry

  • The relational coordinates collect the three pairwise cosine similarities among normalized text, video, and audio embeddings.These similarities form the coordinates used to describe relational geometry.Source: A. Multimodal Relational Geometry · p. 4
  • The analyzed quantity is the Gramian-volume component used in the deployment scoring path, not a standalone retrieval score.It is one component of the combined HyperGRAM score.Source: A. Multimodal Relational Geometry · p. 4
  • In the frozen cohorts, evaluated determinants are nonnegative and the analyzed volume is differentiable at all observed clean points.The score is expressed in three-dimensional relational coordinates, where degradation is represented regardless of whether pixels or waveforms were perturbed.Source: A. Multimodal Relational Geometry · p. 4

B. Perturbation Response Formulation

  • The response is the degraded-state volume minus the clean-state volume, and displacement magnitude is the norm of the change in relational coordinates.Thus, the response formulation measures changes in the geometric score and in its coordinate space.Source: B. Perturbation Response Formulation · p. 4
  • Displacement magnitude D is measured after the frozen modality encoders in relational geometry, not in pixels, waveforms, or embedding space.Different raw perturbation severities can yield similar D, while similar raw severities can yield different D.Source: B. Perturbation Response Formulation · p. 4

C. Directional Geometric Response (DGR)

The first-order Gramian-volume response is a projection of the degradation-induced relational displacement onto the local volume gradient. Its decomposition combines operating-point gain, displacement magnitude, and alignment direction.

C. Directional Geometric Response (DGR)

  • The closed-form gradient supports a first-order expansion at observed clean operating points, while the expansion remains an approximation rather than an identity.The empirical analysis tests whether this first-order term explains observed responses and whether direction is necessary.Source: C. Directional Geometric Response (DGR) · p. 4
  • The first-order response combines local geometric gain, displacement magnitude D, and alignment cos θ between the displacement and the local gradient.The interpretation is that response depends jointly on where the representation operates, how far it moves, and in which direction.Source: C. Directional Geometric Response (DGR) · p. 4, Figure 2 · p. 4
  • DGR requires the observed degraded-state displacement, so it is an explanatory first-order quantity rather than a deployment-time predictor, reliability estimator, or calibration variable.Computing it requires applying degradation and passing the degraded input through the frozen encoder.Source: C. Directional Geometric Response (DGR) · p. 4

D. Datasets and Controlled Perturbations

The protocol uses fixed cohorts from two video–text retrieval benchmarks and controlled single-modality degradations. It evaluates scalar candidates with held-out variance explanation, magnitude-matched rankings, and pre-specified inferential and audit rules.

D. Datasets and Controlled Perturbations

  • MSR-VTT uses 878 samples after excluding 116 videos without usable audio and 6 silent tracks, while DiDeMo uses 980 after excluding 5 videos without audio and 19 silent tracks.The MSR-VTT cohort is fixed by an immutable identifier list; ActivityNet is discussed as a possible external test but not used to broaden the mechanism claim.Source: D. Datasets and Controlled Perturbations · p. 5
  • Video receives Gaussian blur at σ ∈{1, 2, 4} and audio receives additive white Gaussian noise at SNR ∈{20, 10, 5, 0} dB, one modality at a time.Text remains clean, and the four dataset–degradation cells aggregate all tested severity levels on each axis.Source: D. Datasets and Controlled Perturbations · p. 5

E. Evaluation Protocol and Decision Rules

1) Out-of-Sample Variance Explained:

  • Pooled five-fold out-of-sample R2 measures held-out prediction of |∆V| from each candidate scalar using training-fold ordinary least squares with an intercept.Candidates share the same seeded fold partition, and negative R2 means worse than predicting the training mean.Source: 1) Out-of-Sample Variance Explained · p. 5
  • Matched-D ranking tests whether a candidate identifies the larger response within pairs drawn from narrow displacement-magnitude bins, with chance at 0.5.The primary bin width is ε = 0.01; widths 0.005 and 0.02 are sensitivity checks.Source: 2) Matched-Magnitude Pairwise Ranking · p. 5

3) Bootstrap Intervals and Binding Gates:

  • The gain-normalization audit uses 500 video-clustered bootstrap resamples, 5 folds, seed 3105, and pre-specified predictability and clean-ranking gates.The clean-ranking gate uses a 0.98 threshold, and the decision rules were not modified after outcomes were observed.Source: 3) Bootstrap Intervals and Binding Gates · p. 5

IV. PERTURBATION MAGNITUDE IS NOT SUFFICIENT

Across the tested dataset–degradation cells, displacement magnitude explains little of the geometric response, and magnitude-matched samples respond differently. The tested scalar reliability proxies also fail their predictive or matched-magnitude gates.

A. Cross-Condition Magnitude–Response Analysis

  • R² values for D → |∆V| are 0.146, −0.004, 0.136, and 0.021 across the four dataset–axis cells, so magnitude explains at most 15% of response variance.The magnitude-only account is effectively at chance on both audio axes.Source: A. Cross-Condition Magnitude–Response… · p. 5
  • At σ = 4 blur, mean ∆V is +0.00375 but 401 of 878 samples respond negatively; at 0 dB SNR, mean ∆V is +0.01437 but 209 of 878 respond negatively.Samples near the mean displacement can lie on either side of zero response.Source: A. Cross-Condition Magnitude–Response… · p. 5

B. Matched-Magnitude Analysis

  • Within narrow magnitude bins, D ranks the larger response near chance, while samples with nearly equal displacement can differ in response magnitude and sign.Fig. 3 compares absolute volume response with relational displacement magnitude across both cohorts and degradation axes.Source: B. Matched-Magnitude Analysis · p. 5, Figure 3 · p. 6

A. Context-Based Reliability Proxy

  • S1, a geometric operating-point context signal for degraded-state compromise, produces no stable cross-cell explanation under either predictive or matched-magnitude evaluation.Source: A. Context-Based Reliability Proxy · p. 5

B. Local Self-Sensitivity Proxy

  • S2 and S3 remain near chance at 0.460–0.505 on the two MSR-VTT axes, with negligible regression increments; the proxy protocol was run only for that cohort.Source: B. Local Self-Sensitivity Proxy · p. 6
  • These findings concern the tested scalar proxies for this geometric score under the tested perturbations, not whether modality reliability matters generally.The response also depends on the relational direction of displacement.Source: B. Local Self-Sensitivity Proxy · p. 6

C. Direction-Free Geometric Gain

Direction-free geometric summaries discard information that distinguishes responses to similar displacements. The first-order DGR term, which retains displacement direction relative to the local gradient, explains the observed response substantially better.

C. Direction-Free Geometric Gain

  • The gain alone ranks matched-D responses at 0.568 for MSR-VTT video, below the 0.55 gate by its 0.540 lower bound, and collapses to 0.486 on audio.The ∥∇V∥D product passes the gate in only one of four cells.Source: C. Direction-Free Geometric Gain · p. 6

D. Why Scalar Reliability Does Not Explain the Response

B. DGR as the Direction-Aware Response Quantity

  • DGR is an explanatory first-order quantity using the observed degraded-state displacement, not a deployment-time predictor.Source: Table I · p. 7

A. First-Order Local Expansion

B. DGR as the Direction-Aware Response Quantity

  • The first-order DGR magnitude achieves out-of-sample R² of 0.838–0.969, versus at most 0.224 for the strongest scalar alternative.It ranks larger responses within matched-D pairs at 0.864–0.963 accuracy and recovers response sign at 0.909–0.989 accuracy.Source: B. DGR as the Direction-Aware Response… · p. 7

C. Cross-Dataset and Cross-Modality Evidence

Across both datasets and degradation axes, directional response organizes geometric-score changes better than displacement magnitude or the direction-free gain–magnitude product. The first-order approximation remains imperfect, especially for audio, and gain normalization is treated as an audited intervention rather than a proposed method.

C. Cross-Dataset and Cross-Modality Evidence

  • Across all four dataset–degradation cells, magnitude and the direction-free product are weak, while the directional projection is strong.The pointwise comparison supports DGR as the dominant first-order response structure, without implying exact equality.Source: C. Cross-Dataset and Cross-Modality… · p. 7, Figure 4 · p. 8

D. First-Order Approximation Error and Higher-Order Effects

  • Median relative deviation between ∆V and DGR is approximately 7% on video axes and approximately 39–41% on audio axes.Higher-order effects remain non-negligible, particularly for stronger audio perturbations.Source: D. First-Order Approximation Error and… · p. 7

E. Determinant and Volume Levels

  • The same mechanism can be expressed at determinant level, but volume-level results make explicit the quantity used as the score.The clean-volume coefficient of variation is 5.4%–5.8%, so 1/(2Vc) changes determinant-level fits only modestly.Source: E. Determinant and Volume Levels · p. 7

A. Motivation for Gain Normalization

  • The first-order decomposition motivates testing division by the local gradient norm, but the section frames this as a pre-specified negative-result audit, not a normalization method.The audit is intended to establish the boundary of the interpretation rather than introduce a calibration technique.Source: A. Motivation for Gain Normalization · pp. 7–8

B. Pre-Specified Audit Protocol

The pre-specified audit tests gain normalization against response-predictability and clean-ranking gates. The candidate fails both: it does not reliably improve predictability and does not preserve clean top-1 ordering at the required threshold.

B. Pre-Specified Audit Protocol

  • The candidate uses ε anchored to 0.1 times median gV from a pre-specified grid, with no validation-set tuning.The predictability gate requires improvement in at least three of four cells; the clean-ranking gate sets a top-1 keep-rate threshold of 0.98.Source: B. Pre-Specified Audit Protocol · p. 8

C. Predictability Gate

  • Only MSR-VTT video shows a confidence-interval-excluded R² improvement, with ∆R² ∈ [0.0014, 0.0290] and out-of-sample gain +0.028.The other three cells show no reliable improvement, so the requirement of improvement in at least three of four cells is unmet.Source: C. Predictability Gate · p. 8

E. Negative Result and Interpretation

D. Clean-Ranking Safety Gate

  • At ε = 0.1 times median gain, clean top-1 ordering is preserved for 0.9715 of MSR-VTT queries and 0.9296 of DiDeMo queries, below the 0.98 threshold.The anisotropic variant has lower top-1 keep rates: 0.485 for MSR-VTT and 0.386 for DiDeMo.Source: D. Clean-Ranking Safety Gate · pp. 8–9

A. What the Results Establish

Across the tested datasets and degradation axes, response depends on the clean operating point, displacement magnitude, and displacement direction; the tested gain-normalization family fails its pre-specified gates. The evidence is limited to the tested geometry, datasets, and perturbations, and DGR is explanatory rather than a deployment-time predictor.

A. What the Results Establish

  • The gain-normalization candidate fails both gates: predictability improves in only one of four cells, while clean top-1 order retention misses the 0.98 threshold on both datasets.The anisotropic variant’s clean-order rate collapses.Source: Figure 6 · p. 10

B. What the Results Do Not Establish

  • The analysis does not establish causality or universality beyond MSR-VTT, DiDeMo, Gaussian blur, additive white Gaussian noise, and the tested perturbation strengths.Because DGR uses observed degraded-state displacement, its high R2 does not make it a deployment-time predictor.Source: B. What the Results Do Not Establish · p. 10

C. Implications for Multimodal Geometric Representation Learning

  • Future evaluations may report displacement direction alongside severity, since scalar severity summaries can hide directional variation in geometric-score responses.The authors cautiously suggest that future methods may need to consider local directional geometry.Source: C. Implications for Multimodal… · p. 10

D. Limitations and Scope

  • The study covers two datasets, two perturbation families, and one Gramian-volume geometry; first-order approximation is less accurate under stronger audio perturbations.Median relative deviations are roughly 39–41%; DGR also requires displacement observed after degradation.Source: D. Limitations and Scope · p. 10

A. What the Results Establish

  • Magnitude alone explains little of the response; the first-order DGR captures direction-dependent heterogeneity across the tested datasets and perturbation axes.The response is jointly structured by the clean operating point, displacement magnitude, and displacement direction.Source: A. What the Results Establish · pp. 9–10, §IX · p. 11

IX. CONCLUSION

  • Geometric perturbation response depends on operating point, movement magnitude, and direction; the tested normalization does not justify turning this mechanism into score-side calibration.The negative audit applies to the pre-specified V/(gV + ε) family, not every normalization method.Source: §IX · p. 11

APPENDIX A COHORT CONSTRUCTION AND QUALITY CONTROL

The appendices fix the cohorts and perturbation procedures, verify that diagnostic and deployment geometries agree numerically, and specify the scalar proxies used in the audit.

APPENDIX A COHORT CONSTRUCTION AND QUALITY CONTROL

  • The frozen cohorts contain 878 MSR-VTT videos and 980 DiDeMo videos, with audio-quality exclusions applied before response analysis.No sample is removed after observing a response statistic.Source: Appendix A · p. 11

APPENDIX B PERTURBATION DEFINITIONS

  • Video blur uses σ ∈ {1, 2, 4}, while audio noise uses stated signal-to-noise ratios; each condition perturbs one modality with others unchanged.Relational displacement is computed after frozen encoders and pairwise cosine construction.Source: Appendix B · p. 11

APPENDIX C GAIN-NORMALIZATION AUDIT DETAILS

  • The isotropic audit varies ε around median(gV), and the anisotropic covariance variant does not rescue either pre-specified gate.The audit uses 500 bootstrap resamples, five folds, and seed 3105.Source: Appendix C · p. 11

APPENDIX D NUMERICAL GEOMETRY EQUIVALENCE

  • Deployment-form and normalized-form determinants differ by at most 5.0×10^-7 on MSR-VTT and 4.5×10^-7 on DiDeMo, supporting numerical geometry equivalence.Feature norms deviate from one by at most 1.2 × 10^-7.Source: Appendix D · p. 11

APPENDIX E RELIABILITY PROXY DEFINITIONS AND EVALUATION

  • Scalar candidates are computed on clean frozen inputs using probes and views fixed a priori, without oracle degradation levels or response quantities.The context proxy uses clean similarities and volume; local sensitivity uses mild probes, and the geometric gain is the clean local volume-gradient norm.Source: Appendix E · p. 11
  • The self-consistency proxy compares deterministic benign views: resize-and-crop variants for video and opposite 0.25-second shifts for audio.The scalar-proxy statistics are reported only for MSR-VTT because that evaluation used its cohort.Source: Appendix E · p. 11

Related papers

Loading 2610.08533v1…