PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning

Zhaolu Kang, Meixin Wu, Yu Xue, Yingjie He, Qiming Shi, Lei Wei, Yidi Wang, Richeng Xuan, Zhichao Hu

arXiv:2608.29278v1cs.CL

TL;DR

Clean omni-modal benchmarks do not establish stable cross-modal reasoning because intact, redundant inputs can support shortcuts. SCEVAL preserves the task and channels while structurally corrupting text, vision, and audio across a human-verified benchmark. Structural corruption reduces performance, text–vision damage forms the most stable shared fault line, and joint degradation is non-additive, showing that clean accuracy alone is insufficient to characterize reliability under structural evidence damage.

  • Problem

    Clean evaluation cannot distinguish stable cross-modal binding from success based on intact single-channel cues, while missing-modality ablations change the task.

  • Method

    SCEVAL keeps the question, answer space, and text, vision, and audio channels present while applying controlled structural corruptions individually and jointly.

  • Results

    Structural corruption reduces performance relative to clean inputs; text–vision damage is the most stable shared fault line, and joint degradation is not explained by corruption count.

  • Takeaways & Limitations

    Clean omni-modal accuracy alone is insufficient to characterize how reliably models use evidence across modalities when that evidence becomes structurally unreliable.

  • Takeaways & Limitations

    SCEVAL covers 273 English-language tri-modal examples, uses curated non-adversarial operators, evaluates structured multiple-choice decisions, and leaves causal representation explanations open.

Abstract

from arXiv · show

Omni-modal large language models are increasingly evaluated on clean text--vision--audio inputs, where every channel is present, synchronized, and readily interpretable. Such scores are often taken as evidence of robust cross-modal fusion, but clean evaluation cannot tell whether success depends on stable cross-modal structure or on cues sufficient only in intact inputs. To address this gap, we define a modality fault line: a boundary at which model behavior becomes unstable when a modality remains present and human-interpretable, but its internal evidence structure is perturbed. We introduce SCEval (Structure-Corruption Evaluation) a diagnostic evaluation protocol that keeps the question, answer space, and modality channels fixed while applying controlled structural corruptions to text, vision, and audio individually and jointly. Built from $273$ human-verified tri-modal examples from Social-IQ, OmniBench, and VALOR, SCEval evaluates $15$ proprietary and open-source omni-modal systems. The results show that structural corruption lowers clean accuracy, text--vision damage forms the most stable shared fault line, and multi-modal degradation is non-additive rather than a simple function of the number of corrupted modalities. Clean omni-modal accuracy therefore does not establish that a model will remain reliable when cross-modal evidence becomes structurally unreliable.

1 Introduction

Clean omni-modal accuracy can hide reliance on intact, redundant evidence rather than stable cross-modal binding. SCEVAL probes the resulting modality fault lines by preserving channels while corrupting their internal structure.

  • Motivation: Clean evaluation cannot distinguish cross-modal binding from single-channel cues or shortcuts when all evidence streams are well formed.Missing-modality ablations instead remove a channel, changing the task and conflating reliance with distribution shift.
  • Motivation: A modality fault line is a hidden boundary where competence breaks while a modality remains present, interpretable, but structurally unreliable.The diagnostic targets noisy, fragmented, or otherwise damaged internal evidence rather than absent channels.
  • Benchmark: 273 human-verified tri-modal examples are constructed from Social-IQ, OmniBench, and VALOR under a unified annotation scheme.The benchmark includes clean inputs and single-, dual-, and tri-modal corruption variants.
  • Protocol: SCEVAL keeps the task and modality channels fixed while applying structural corruption to text, vision, and audio individually and jointly.This design separates structural fragility from missing-modality effects.
  • Findings: Structural corruption lowers clean performance, text–vision damage is the most stable shared fault line, and joint degradation is non-additive and model-dependent.The reported pattern is not explained by simply counting corrupted modalities.

2 Structure-Corruption Evaluation

SCEVAL evaluates structural fragility by corrupting internal evidence while preserving the task, channels, and gold answer. Its paired benchmark and aggregation procedures support clean-versus-corrupted comparisons across modalities and corruption combinations.

  • Protocol: SCEVAL preserves the question, answer options, modality containers, and gold label while damaging internal text, visual, or audio evidence.Combined conditions apply one, two, or three operators to the same base example.
  • Benchmark construction: The benchmark combines three omni-modal sources spanning social reasoning, joint text–image–audio reasoning, and naturally co-occurring visual and acoustic events.The sources are Social-IQ, OmniBench, and VALOR.
  • Benchmark construction: 273 examples remain from a 300-candidate pool after third-party checks of validity, tri-modal answerability, and corrupted-variant interpretability.The released manifest supports reconstruction and auditing of the filtering rule.
  • Evaluation protocol: Accuracy for stochastic conditions is averaged across up to three random variants, with missing variants excluded from accuracy estimation.Unparseable responses are treated as missing for accuracy estimation and incorrect for significance testing.
  • Evaluation protocol: The protocol reports clean-baseline, worst-single, fault-line, misleading-modality, fragility-slope, and shortcut-gap diagnostics.These measures isolate performance loss, compound failure, modality effects, severity sensitivity, and linguistic-channel shortcuts.
  • Evaluation protocol: Figure 3 illustrates sample-paired evaluation with clean and corrupted variants that retain the gold answer and all modality channels.This distinguishes the probe from missing-modality ablations.

3 Experimental Results

SCEVAL evaluates 15 omni-modal systems under controlled single- and joint-modality structural corruptions. Results reveal concentrated operator fragility, coverage-sensitive severity effects, and structured non-additive cross-modal degradation.

  • 15 systems were evaluated with standardized multiple-choice prompts, mean accuracy over up to three stochastic variants, coverage, invalid-output rates, and bootstrap intervals.
  • Five operators produced panel-mean drops of at least 7 pp, while the remaining nine stayed below 5 pp.The severe group comprised drop_words, word_shuffle, noise, remove, and mute; cosmetic operators often stayed near clean performance.
  • Severity curves accelerated past 30 for drop_words, word_shuffle, and noise across all 15 systems, while other operators remained within 1–3 pp of clean.Figure 4 averages three stochastic variants, so trends are more informative than individual points.
  • Joint degradation was non-additive: the worst cell paired mildly corrupted text t30 with heavily corrupted vision v70, while trimodal damage tracked vision severity more than corruption count.Text operator identity caused larger model-specific swings for Qwen than Gemini; vision and audio substitutions shifted accuracy by at most a couple of percentage points.
  • Clean tri-modal accuracy shows that an answer can be found with intact evidence, but not that the same evidence use survives structural degradation.SCEVAL keeps every channel present and human-interpretable, distinguishing structural fragility from missing-modality distribution shift.

4 Related Work

Prior work evaluates multimodal robustness through single-modality corruption and related multimodal perturbation suites. SCEVAL is positioned as extending this line by testing cross-modal structural relations rather than only individual channels.

  • Omni-modal systems are commonly evaluated on clean tri-modal question-answering suites such as Social-IQ and OmniBench.
  • Existing corruption benchmarks span vision, audio, and multimodal settings, including ImageNet-C, MM-Robustness, MMCBench, and MMMU perturbations.
  • These multimodal benchmarks corrupt one modality at a time and do not test the cross-modal relation.SCEVAL is contrasted with these approaches along six design axes.

5 Conclusion

SCEVAL shows that omni-modal models are not equally robust to structural degradation across modalities. Clean benchmark accuracy alone is insufficient to characterize reliable evidence use across modalities.

  • Structural corruption reduces performance relative to the clean all-modality baseline across the reported model panel.
  • Text–vision corruption emerges as the most stable bimodal fault line.
  • Degradation is not explained by a simple count of corrupted modalities.

6 Limitations

SCEVAL is a controlled diagnostic rather than an exhaustive robustness census, with important limits in coverage, interpretability, and evaluation scope. Its results should therefore be read as conservative and structured-setting evidence.

  • SCEVAL evaluates 273 verified tri-modal examples from three English-language benchmarks rather than providing an exhaustive robustness census.Joint-corruption analysis focuses on four deep-dive models, while other systems use a lightweight panel.
  • Rejected uninterpretable variants are concentrated in occlusion, brightness, mute, and remove, making corresponding severity-70 results conservative estimates.Claims that corruptions improve performance are restricted to six confound-free operators.
  • The protocol measures multiple-choice JSON decisions with static human-readable corruptions rather than open-ended or adversarially optimized behavior.The reported failures are therefore lower-bound estimates of possible corruption sensitivity.

Ethics Statement

The work studies robustness under controlled input corruptions before deployment in noisy multimodal environments. It does not introduce new user data beyond source examples, and derived assets must respect underlying data constraints.

  • The benchmark studies model failure modes under controlled input corruptions before deployment in noisy multimodal environments.
  • The benchmark introduces no new user data beyond the evaluated source examples.
  • Any release of derived assets should respect the licences and privacy constraints of the underlying data.

B Benchmark Comparison Details

The detailed analyses connect operator-level severity patterns to the headline joint-corruption findings. They show why text–vision is the dominant shared fault line and identify the operators driving modality-specific damage.

  • The comparison positions SCEVAL against clean tri-modal QA, clean-modality shortcut suites, and single-modality corruption benchmarks.
  • Each joint-corruption table row reports mean-variant accuracy and absolute change from the model’s clean all-modality baseline.
  • Text word dropping, visual additive noise, and audio segment muting or removal are the most damaging operators for their respective modalities.Audio distortion is consistently the weakest audio operator.
  • The strongest joint condition combines light text damage with heavy visual noise, explaining why text–vision becomes the dominant joint fault line.Text–audio and vision–audio combinations are less stable as shared fault lines because audio distortion alone is weak.

C.3 Does combined corruption exceed the weakest single modality?

Combined corruption usually does not exceed the weakest corresponding single-modality corruption, but it consistently reduces accuracy relative to the clean baseline and can exceed that weakest degradation in a few high-pressure conditions.

  • For Gemini 3 Flash, all 29 joint cells remain above the weakest single-modality cell, so none exceeds the weakest single-modality degradation.The weakest single-modality reference is drop_words at severity 70.
  • For Qwen3.5-Omni-Plus, 26 of 29 joint cells stay above the weakest single-modality cell, while three narrowly cross the threshold.All three exceptions combine drop_words with noise under heavy visual severity; the largest exceedance is +1.09 pp.
  • Combined corruption consistently lowers performance relative to the clean all-modality baseline, even when it is not worse than the weakest single-modality corruption.This distinction separates absolute degradation from comparison against the weakest component.
  • Across the deep-dive panel, joint corruption is behaviorally sub-additive rather than a simple sum of single-modality drops.Average gaps from the additive prediction range from +0.94 to +22.62 percentage points.

C.5 All-model panel results

The all-model panel aggregates fault-line behavior across modality combinations and diagnostic controls, revealing consistent structural degradation, modality-specific severity patterns, and limited proportional transfer from clean accuracy to robustness.

  • Table 8 summarizes aggregate fault-line scores and strongest observed conditions for the 15-model lightweight panel.The panel uses a fixed 12-cell comparison structure across text–vision, text–audio, vision–audio, and tri-modal blocks.
  • The mildly corrupted-text versus heavily corrupted-text contrast is positive for 14 of 15 evaluated models.This is a fixed-panel descriptive comparison, not evidence that related model versions are independent samples.
  • The panel also includes controls for modality absence, shortcut reliance, source consistency, invalid outputs, and joint-corruption interaction.These controls distinguish structural-dependence diagnostics from missing-modality performance and track coverage and reliability.
  • Alternative median and worst-2 aggregation rules preserve the operator ranking, misleading-modality effect, and sub-additivity pattern.The mean−worst gap is ≲2 pp at every sampled cell, limiting the effect of variant aggregation on qualitative conclusions.
  • Text has the steepest fragility slope for all 15 models, with magnitudes of 0.09–0.16 percentage points per severity unit.Vision slopes are typically shallower, while audio slopes range from near-flat to occasionally approaching text.
  • Joint corruption is evaluated against additive predictions based on matched single-modality drops, with positive interaction values indicating gentler-than-additive degradation.All 16 deep-dive model-by-combination point estimates are positive, although support is heterogeneous.

D Human Filtering and Data Verification Details

Human verification filters source examples and corrupted variants for validity, tri-modal answerability, modality presence, interpretability, and answer preservation before inclusion in the evaluated set.

  • The audit trail documents how the verified base set was derived, including annotation instructions, rejection counts, and reasons.This process supports downstream auditing of source and modality representation.
  • The initial pool contains 300 candidates, with 100 subsampled from each of Social-IQ, OmniBench, and VALOR after oversampling plausibly tri-modal examples.Candidates are normalized into a common record containing the question, answer space, and media evidence.
  • Examples must pass clean validity and tri-modal answerability, including evidence from at least two modalities and accessible, unbroken media.The form also checks modality availability and answer leakage.
  • The protocol separately checks whether the original gold answer remains defensible after corruption, distinguishing robustness tests from stress tests.Human-normalized and gold-preserved analyses use this audit to define their evaluated cohort.
  • The filtering funnel reduces 300 candidates to 285 after clean validity, 277 after leakage filtering, and 273 verified base examples after corruption-interpretability filtering.The final set contains 100 Social-IQ, 77 OmniBench, and 96 VALOR examples.
  • Rejected candidates are recorded with identifiers, sources, failure reasons, and adjudication notes; 23 of 27 rejections come from OmniBench.The concentration is consistent with OmniBench items being more borderline tri-modal after normalization.
  • Audio and vision variants are separately judged for perceptual interpretability, retaining 76.3% of audio cells and 83.2% of vision cells.Occlusion and audio remove/mute operators account for especially concentrated rejections.

E Corruption-Operator Definitions

SCEVAL defines a curated taxonomy of structural operators that preserve each modality while perturbing its internal lexical, spatial, or temporal evidence structure at controlled severities.

  • The fourteen operators are restricted to structural rather than semantic damage, keeping each modality physically present.Meaning-changing substitutions such as replacing a question or visual scene are excluded.
  • Severity values s ∈ {10, 30, 50, 70} map to concrete parameters for each operator, while stochastic variants differ in selected tokens, pixels, or temporal segments.The perturbation magnitude is fixed by severity; random variants change which evidence elements are selected.
  • Every operator preserves the sample pairing and original answer while damaging internal modality evidence rather than substituting the input.The design supports clean-versus-corrupted comparisons on the same example.
  • Operators rejected by human validation at high severity are treated as lower-bound estimates of robustness loss in headline results.This applies to conditions such as occlusion, brightness, mute, and remove when more than half of variants are rejected.

F Experimental Setup Details

The experimental setup standardizes multimodal inputs, model interfaces, corruption conditions, prompting, and aggregation to support comparable fault-line measurements across a 15-model panel.

  • Model panel: SCEVAL evaluates seven proprietary/API and eight open or open-API omni-modal systems under a standardized panel protocol.The headline severity-70 single-modality matrix covers all 15 models and 14 operators; joint-corruption deep dives use four representative systems.
  • Input standardization: The protocol uses identical frame-extracted visual evidence across models, with uniform temporal sampling of up to eight frames per video example.The same ordered frame set is reused across models and corruption conditions, while native-video interfaces are flagged separately.
  • Input standardization: Audio is standardized to mono 16 kHz, with source duration preserved and clean audio aligned to the extracted visual frames.Audio begins at the start of the first extracted frame, giving zero clean-condition audio–visual offset.
  • Inference protocol: All headline runs use one multiple-choice prompt requiring a parseable JSON response, enabling comparable accuracy, invalid-output rate, and variant aggregation.The prompt states that evidence may be structurally degraded while exactly one option remains correct.
  • Aggregation and reporting: Stochastic conditions are aggregated at the original-sample level by averaging correctness across up to three random variants, with clean-baseline drops and validity statistics reported.Worst-variant accuracy is retained as a seed-variance diagnostic, alongside effective trial counts where possible.
  • Evaluation design: The evaluation is organized as complementary experiment families, each specifying its manipulated input, comparison baseline, reported metric, and interpretation boundary.The matrix includes lightweight all-model evaluation, expanded representative-model suites, controls, robustness checks, and mechanism probes.

G.1 A. Clean and single-modality experiments

The clean and corruption experiments retain all modalities while structurally damaging one or more channels, then compare single- and joint-corruption effects using standardized accuracy and diagnostic controls.

  • Clean and single-modality experiments: The clean run establishes the all-modality reference, while single-modality tests damage exactly one channel and retain the others for later comparisons.Combined-corruption drops are compared against the weakest individual component, not only the clean baseline.
  • Single-modality suite: For each of 273 verified examples, one modality is replaced by a corrupted variant while the question, answer, and remaining evidence stay unchanged.Operators target the text, extracted visual frames or image asset, and audio clip respectively.
  • Single-modality suite: The single-modality suite contains 56 cells: 16 text, 28 vision, and 12 audio conditions across four severities.Text uses four operators, vision seven, and audio three, each evaluated at severities 10, 30, 50, and 70.
  • Metrics and interpretation: Single-modality tables report accuracy, ∆base, valid-output coverage, valid trials, and severity curves, while the all-model version adds modality-specific fragility slopes.A large drop identifies a fragile channel but does not by itself establish cross-modal interaction.
  • Combined corruptions: The combined suite tests whether damage across channels produces a fault line, using ∆base for clean-performance loss and ∆single for comparison with the weakest single corruption.Positive ∆single is stronger evidence for a non-additive or interaction-like fault line.
  • Combined corruptions: The 12-cell lightweight panel preserves the main diagnostic structure across all models, while the full expanded suite is reserved for representative systems.It includes the four text–vision cells, two text–audio cells, two vision–audio cells, and four trimodal cells.
  • Controls and results: Across 15 models, the misleading-modality effect averages −4.31 pp for text, +1.87 pp for vision, and +2.61 pp for audio.Positive vision and audio effects occur in 13 of 15 and 12 of 15 models, respectively.
  • Controls and results: The text-only shortcut gap averages +15.57 pp above the 25% random-guess baseline, but remains below the clean accuracy advantage of approximately 45–55 pp.This indicates that text-only language priors account for a minority of clean performance.
Loading 2608.29278v1…