PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models

Laura Yao, Shixing Yu, Jie Peng, Promit Ghosal, Kyra Gan, Tianlong Chen

arXiv:2610.14190v1cs.AI

TL;DR

VLM attribution methods generally identify either relevant visual evidence or influential training examples, while latent-mediated influence lacks the spatial and multimodal structure needed to link them. MOSAIC learns a shared sparse image–text latent space and decomposes training-example influence through localized concepts. Its features are more localized while largely preserving task performance, influence-ranked latents are more behaviorally consequential than activation-ranked ones, and image evidence can substantially reorganize training-example rankings while latent rankings remain more stable.

  • Problem

    VLM attribution methods rarely connect localized visual evidence to the training examples that shaped a prediction, and latent-mediated influence methods lack multimodal and spatial grounding.

    Source: §1 · p. 1

  • Method

    MOSAIC learns a shared sparse image–text latent basis with spatially coherent visual features and phrase–region alignment, then decomposes training-example influence through those latents.

    Source: §1 · p. 2

  • Results

    Across tasks and backbones, MOSAIC yields more localized features while largely preserving task performance; influence-ranked latents are more behaviorally consequential, and removing images can substantially change example rankings while latent rankings remain positively correlated.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    MOSAIC links influential training examples to the multimodal concepts and localized evidence through which they affect predictions.

    Source: §1 · p. 3

    Features may remain partially entangled, attribution depends on layer and dictionary capacity, phrase–region alignment may use noisy or incomplete attention, and EK-FAC influence approximates full retraining effects.

    Source: A Limitations and Future Work · p. 14

Abstract

from arXiv · show

Explaining a vision--language model (VLM) prediction requires tracing training-example influence to the multimodal evidence that mediates it, yet existing attribution methods largely operate only at the sample level. Recent latent-mediated influence methods provide feature-level attribution in language models, but lack the multimodal and spatial structure needed to ground influence in VLMs. We introduce MOSAIC (Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization). It learns a shared sparse image--text latent space whose concepts are 1) sparse within each modality, 2) supported on contiguous image regions, 3) aligned to the phrases that describe them, and 4) approximately decorrelated. It then applies EK-FAC to decompose training-example influence through these concepts and localize their contributions to specific text phrases and image regions. We evaluate MOSAIC on Qwen and Gemma across captioning, chest X-ray understanding, and visual question answering. The learned features are more localized and coherent without degrading task behavior, and ablating influence-ranked latents changes model outputs more than ablating latents ranked by activation or frequency. We find that visual evidence can substantially reorganize training-data attribution while preserving more of the shared latent structure. MOSAIC thus connects where model behavior comes from with what multimodal evidence carries that influence.

1 Introduction

MOSAIC addresses the gap between locating visual evidence and identifying the training examples that shaped a VLM prediction. Its shared sparse latent basis links influential examples to multimodal concepts and localized phrases and image regions, with experiments showing localized features, behaviorally consequential influence rankings, and image-dependent changes in example attribution.

1 Introduction

  • Existing approaches typically localize prediction-relevant regions or identify influential training examples, but do not connect both to the specific concepts within an example that drove a prediction.Latent-mediated influence had been developed for language models, while multimodal SAEs exposed image–text features without tracing their training origins.Source: §1 · pp. 1–2
  • MOSAIC learns a shared TopK image–text dictionary with spatially coherent visual features, phrase–region alignment, and decorrelated latent activity, then decomposes example influence through those features.Its EK-FAC estimator projects feature contributions back to text phrases and image regions.Source: §1 · p. 2
  • On GQA with Qwen, visual footprint drops from 0.688 to 0.120 while task performance is largely preserved.The footprint is the fraction of image patches on which a feature activates; similar localization gains occur across tasks and backbones.Source: §1 · p. 2
  • On GQA with Gemma, removing the top 256 influence-ranked latents flips 38.3% of predictions versus 8.3% for activation-ranked latents.The same trend is reported across datasets and models.Source: §1 · p. 2
  • Removing the image yields training-example rank correlation as low as ρ = −0.797, while latent rankings remain positively correlated at up to ρ = 0.795.The influential examples can change substantially while more of the underlying latent feature structure remains preserved.Source: §1 · pp. 2–3

2 Related Work

Prior work studies training-example influence, visual evidence, and sparse multimodal representations, but does not bring them together to connect VLM predictions to training examples and localized cross-modal evidence.

2 Related Work

  • Training-data attribution estimates how examples affect predictions, and latent-mediated influence decomposes that effect through SAE features, but the latter was developed for language models.MOSAIC extends the latent view to linguistic and spatially organized visual representations.Source: §2 · p. 3
  • Visual and multimodal attribution methods identify or manipulate prediction-relevant evidence but do not connect it to the training examples that shaped a prediction.MOSAIC uses frozen VLM attention for phrase–region correspondence, while deriving attribution from training-data influence through learned latents.Source: §2 · p. 3
  • Existing multimodal sparse representations are not designed to mediate example-level training-data influence; MOSAIC decomposes influence through a position-resolved image–text latent basis.The basis combines spatial coherence and phrase–region alignment to connect predictions, training examples, concepts, and localized evidence.Source: §2 · p. 3

3 Method

MOSAIC builds a shared sparse image–text basis with spatial and phrase–region structure, decomposes example influence over localized features, and uses EK-FAC to make curvature-corrected influence estimation scalable.

3 Method

  • MOSAIC first learns a reconstructive SAE dictionary, then shapes it with spatial and cross-modal objectives before decomposing influence through its features.The method estimates contributions at latent, phrase, and image-region or patch levels.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Background and Multimodal Latent Basis

  • A shared position-wise TopK SAE encodes image and text positions with a common dictionary, while decorrelation regularizes latent activity rather than enforcing decoder orthogonality.TopK retains at most k active features per position, and the decoder reconstructs each intermediate hidden state.Source: §3.1 · pp. 3–4

3.2 Spatially Structured Visual Sparsity

  • Because token sparsity alone does not ensure coherent visual support, MOSAIC clusters patches into regions and applies group and exclusive sparsity to concentrate each latent on few coherent regions.Regional support and exclusivity are normalized to reduce sensitivity to region size and partition granularity.Source: §3.2 · pp. 4–5

3.3 Phrase–Region Alignment

  • Phrase–region alignment pools pre-activations, uses frozen VLM attention to select each phrase’s positive region, and contrasts it against other regions in the same image.The cross-modal objective also includes magnitude-mismatch and modality-balance terms.Source: §3.3 · p. 5

3.4 Multimodal Influence Decomposition

  • Position-wise latent attribution maps a training example’s influence to individual text tokens and image patches, which can be grouped into phrases or spatial regions.This organizes influence as a chain from prediction to training example to sparse latent to localized linguistic and visual evidence.Source: §3.4 · p. 5

3.5 Scalable Influence Estimation with EK-FAC

  • EK-FAC avoids constructing the full curvature matrix by rotating layer gradients into a structured eigenbasis, applying damped inverse-curvature rescaling, and rotating them back.Curvature statistics are estimated once and reused across training–test pairs; examples are ranked by the magnitude of signed influence.Source: §3.5 · p. 6

4 Experiments

Experiments evaluate task-adapted Qwen and Gemma models across VizWiz captioning, CheXpert classification, and GQA question answering. MOSAIC yields grounded, behaviorally meaningful latents and shows that visual evidence changes which training examples matter more than which latent concepts carry their influence.

4.2 Grounded Multimodal Representation

  • Structured sparsity and phrase–region alignment contribute complementary benefits, and combining them yields the strongest overall latent representation.The combined objective has footprint 0.083, contiguity 0.548, balance 26.7%, MS Lift 0.0214, and grounding 0.084.Source: §4.2 · p. 7, Table 2 · p. 7
  • Visual footprint falls from 0.688 to 0.120 on GQA with Qwen, while the SAE changes task performance only marginally across datasets and backbones.The same localization trend appears on VizWiz with Gemma, and CheXpert examples show sharper visual support and more relevant radiological terminology.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Validating Latent Influence: Necessity and Sufficiency

  • On GQA with Gemma, removing 256 influence-ranked latents flips 38.3% of predictions, versus 8.3% for activation-ranked latents; with 64 latents, sufficiency is 75.0% versus 48.3%.The necessity and sufficiency trends also hold on VizWiz and CheXpert.Source: §4.3 · pp. 7–8

4.4 How Visual Evidence Shapes Training-Data Influence

  • Across six model–dataset settings, latent influence correlations remain positive at ρ = 0.543–0.795, while training-example correlations range from −0.797 to −0.564 after image removal.The results suggest visual evidence more strongly conditions which training examples support a prediction than the latent concepts carrying influence.Source: §4.4 · pp. 8–9, Figure 8 · p. 8

4.5 Hierarchical Multimodal Attribution

  • For a VizWiz example, the attribution hierarchy links influential training examples to medicine-, bottle-, and wooden-surface evidence localized in both phrases and image regions.A complementary query-conditioned view ranks medicine- and bottle-related latents on corresponding image regions.Source: Figure 4 · p. 7, §4.5 · p. 9, Figure 9 · p. 9

4.6 Attribution for Correct and Incorrect Predictions

  • In representative GQA cases, grounded influence aligns with relevant visual evidence for a correct answer but traces an incorrect answer to a salient concept supporting the mistake.The comparison illustrates how the decomposition can diagnose what evidence supports a prediction and why that support matters.Source: §4.6 · pp. 9–10, Figure 10 · p. 10

5 Conclusion and Discussion

MOSAIC traces VLM predictions from influential training examples through sparse multimodal concepts to localized linguistic and visual evidence. Across two VLM families and three tasks, its features preserve model behavior while improving localization and semantic coherence, and visual evidence reshapes training-example attribution more than latent structure.

5 Conclusion and Discussion

  • MOSAIC links influential training examples to localized phrases and image regions through a shared sparse multimodal latent basis.The framework combines spatially coherent visual structure, phrase–region alignment, and latent-mediated influence.Source: §5 · p. 10
  • Across two VLM families and three tasks, learned features preserve model behavior while becoming more localized and semantically coherent.Influence-ranked latents are more behaviorally consequential than activation- or frequency-based alternatives.Source: §5 · p. 10
  • Visual evidence can strongly reorganize which training examples matter while preserving substantially more of the underlying latent structure.The authors describe this as a path toward finer-grained and more auditable training-data attribution in multimodal models.Source: §5 · p. 10

A Limitations and Future Work

MOSAIC’s conclusions depend on the quality of its learned SAE representation and the approximation used for influence.

A Limitations and Future Work

  • Latent features may remain partially entangled, limiting how cleanly the learned representation separates concepts.The approach also depends on the chosen layer and dictionary capacity.Source: A Limitations and Future Work · p. 14
  • Phrase–region alignment relies on pretrained attention as a correspondence signal, which may be noisy or incomplete.Source: A Limitations and Future Work · p. 14
  • EK-FAC and latent-mediated influence approximate the effect of full retraining rather than measuring it directly.The authors identify stronger validation against retraining-based influence as future work.Source: A Limitations and Future Work · p. 14

B Curvature Approximation Study

The curvature study compares scalable approximations for representation-mediated influence and adopts EK-FAC after finding alternatives unstable or geometrically uninformative in this setting. Controlled small-scale checks support EK-FAC’s fidelity while it remains tractable at model scale.

B Curvature Approximation Study

  • Raw-Hessian conjugate gradients encounter negative curvature and terminate prematurely, producing influence estimates that vary with the optimization trajectory.Solutions from 10 and 160 iterations share no examples among their respective top-10 training examples.Source: B Curvature Approximation Study · p. 15
  • Generalized Gauss–Newton is positive semidefinite but its damped systems remain difficult and expensive to solve, making it impractical at the required scale.Source: B Curvature Approximation Study · p. 15
  • With 32 curvature examples in an approximately 6 × 10^8-dimensional parameter space, near-orthogonal cohort gradients cover very few test-gradient-relevant directions.Doubling cohort size increases captured gradient mass by only approximately 1.3×.Source: B Curvature Approximation Study · p. 15
  • The empirical Fisher’s CurvAlign is 0.987, indicating that its curvature transformation leaves the test-gradient direction nearly unchanged and behaves like rescaled first-order gradient similarity.The authors attribute this to damping dominating most directions in a low-rank estimate.Source: B Curvature Approximation Study · pp. 15–16
  • Accurate low-rank solves and stable rankings do not establish that empirical-Fisher curvature is informative; its transformation can remain nearly unchanged by damping.The reported Woodbury residual ratios are approximately 10^-6 and rankings across disjoint cohorts have Spearman ρ = 0.993, yet the curvature remains geometrically weak.Source: B Curvature Approximation Study · pp. 15–16
  • EK-FAC achieves cosine fidelity of 0.997–0.9996 across dense, ragged, and masked conditions in controlled settings, supporting its use for the paper’s influence results.It models layerwise activation and gradient statistics, avoiding the severe rank restriction of cohort-based estimates while remaining tractable at model scale.Source: B Curvature Approximation Study · p. 16

C Full VizWiz Captioning Results

On VizWiz captioning, LoRA finetuning substantially improves performance while SAE insertion causes only minor metric changes. The supplementary evaluation defines diagnostics for representation quality, including spatial localization and semantic coherence.

C Full VizWiz Captioning Results

  • LoRA finetuning substantially improves VizWiz captioning over prompted and unprompted base models, while SAE insertion causes only minor changes across metrics.The reported results indicate strong preservation of task-relevant information after SAE insertion.Source: C Full VizWiz Captioning Results · p. 16

D Additional Representation Evaluation

D.1 Latent Quality Metrics

  • A latent is dead if it never activates on the 1,500 evaluation examples, and modality balance uses token-normalized activation rates to avoid sequence-length dominance.A live latent is considered balanced when its modality score is between 0.35 and 0.65.Source: §D.1 · p. 17
  • Visual footprint measures the fraction of an image patch grid occupied by a latent, while contiguity compares its connected-component count with a size-matched random mask.Contiguity values below 1 indicate support more spatially contiguous than chance; it supplements rather than replaces footprint.Source: §D.1 · p. 17
  • Visual monosemanticity compares latent-active patches with a size-matched random support, and the main text reports null-corrected lift rather than raw MS.Latents whose median footprint covers 25% or more of the patch grid are excluded because the comparison becomes degenerate near full-image coverage.Source: §D.1 · p. 17
  • Visual MS is omitted for CheXpert because stereotyped global anatomy makes the assumption that a meaningful region is more self-consistent than its surroundings poorly matched to radiographs.Source: §D.1 · pp. 17–18
  • Causal grounding measures the change in negative log-likelihood for a latent’s top-associated word when that latent is ablated at its active image patches.Examples with image dependence below 0.05 and latents active on fewer than two image patches are excluded; the mean effect is reported with a solo-effect p-value.Source: §D.1 · p. 18

D.2 Latent-Space Orthogonality

The SAE substantially reduces coordinate correlations across VizWiz and CheXpert, yielding a less-correlated basis for per-latent influence analysis. The remaining sections describe layer sensitivity, retrieval, computational scope, implementation, and training choices.

D.2 Latent-Space Orthogonality

  • 99.5% of VizWiz and 99.2% of CheXpert coordinate pairs are near-orthogonal after SAE encoding, versus 20.8% and 25.4% before encoding, respectively.Mean absolute off-diagonal correlation falls from 0.280 to 0.0020 on VizWiz and from 0.239 to 0.0036 on CheXpert. Stable rank is supplementary because the compared spaces have different widths; the text representation is only a reference.Source: §D.2 · pp. 18–19, Table 5 · p. 19

D.3 Sensitivity to SAE Layer Placement

  • Moving Gemma’s SAE from layer 19 to layer 12 reduces GQA accuracy to 0.3315, approximately 26 points below the primary placement and below base performance.The main Gemma experiments therefore use layer 19, where the SAE preserves downstream behavior.Source: §D.3 · p. 19, Table 6 · p. 19

E Gradient-Based Candidate Retrieval

F Computational Scope and Model Scale

  • The full pipeline is limited to approximately 2B-parameter models because per-example gradients, EK-FAC statistics, and latent-resolved influence computation approach a single GPU’s practical memory and compute limits.The authors leave evaluation of larger VLMs to future work.Source: F Computational Scope and Model Scale · p. 19

G Latent-Mediated Influence Implementation

  • Global influence computation reduces derivatives across valid sequence positions, while localization retains position-wise activations to map text contributions to phrases and image contributions to patches and regions.Patch maps support visualization; phrase- and region-level aggregates provide the grouped multimodal decomposition.Source: G Latent-Mediated Influence… · p. 20

G.1 SAE and Decorrelation Details

  • The SAE objective combines reconstruction, decorrelation, auxiliary and load-balancing terms, structured sparsity, and cross-modal alignment; the VLM remains frozen during SAE training.Load balancing discourages activity from collapsing onto a small subset of the dictionary, while decorrelation penalizes off-diagonal activity correlations without forcing decoder orthogonality.Source: §G.1 · pp. 20–21

G.1.1 Additional Phrase–Region Alignment Details

  • The phrase–region objective aligns representations while matching response scale and balancing shared latent participation across modalities.Magnitude matching permits moderate activation-scale differences, and the balance constraint is gated to latents relevant to matched content while allowing some modality specialization.Source: §G.1.1 · p. 21

G.1.2 AuxK Activation for Dead-Latent Recovery

  • Raw signed AuxK activations produce auxiliary reconstructions approximately 7–14× larger than the residual, while a leaky rectifier gives negative inactive latents a recovery signal without changing the positive path.A synthetic check found that ReLU gave persistently negative candidate latents zero auxiliary reconstruction and gradient, whereas the leaky rectifier produced finite reconstruction and nonzero gradients.Source: §G.1.2 · p. 22

H Implementation Details

The implementation uses an 8192-latent TopK SAE and staged training, with task-specific dataset construction and a Gemma placement constraint on patch-level causal grounding. The additional modality analysis examines whether token-count imbalance biases influence.

H Implementation Details

  • Both backbones use an 8192-latent SAE with TopK sparsity k = 128, and inputs are limited to 512 tokens with images resized to at most 512 pixels.The SAE is placed at layer 15 of 28 for Qwen and layer 19 of 35 for Gemma, at approximately matched relative depths.Source: H Implementation Details · p. 22
  • Gemma’s shared upper-layer keys and values prevent patch-level causal grounding at the primary layer-19 SAE placement.The authors state that this architectural property does not affect interventions on all relevant token positions.Source: H Implementation Details · p. 22

H.1 Warm-Start Initialization

  • Stage A learns a sparse dictionary using reconstruction alone, and Stage B adds shaping objectives from that initialization to avoid severe latent collapse.Training Stage B from scratch with strong phrase–region alignment leaves 3,371 latents inactive; 6,000 warm-up steps reduce that count to 142, compared with 56 after full Stage A training.Source: §H.1 · p. 23

H.2 Dataset Processing and Task Construction

  • GQA retains at most one question per image and uses disjoint validation and test subsets; its target places the canonical short answer before the full-answer sentence.Source: §H.2 · p. 23
  • CheXpert balances absent, uncertain, and present labels independently for five findings and holds out validation patients to prevent patient-level leakage.The prompt asks for the status of a finding, and the target begins with the answer letter followed by the cleaned report impression.Source: §H.2 · p. 23
  • VizWiz uses standard captions after removing rejected and precanned responses, while latent necessity and sufficiency analyses target the finetuned model’s own generated caption.Using the model’s output tests whether latent perturbations recover or disrupt its prediction rather than agreement with another human reference.Source: §H.2 · p. 23

I Additional Modality Analysis

I.1 Modality Imbalance

Modality balancing does not improve attribution: curvature-side balancing degrades the reported attribution measure, while representation-gradient balancing barely changes rankings. Removing visual evidence preserves more agreement among latent influences than among retrieved training examples, and qualitative examples show localized, semantically interpretable features across tasks.

I.1 Modality Imbalance

  • 91.3% at α = 0 falls to 77.8% at α = 1 for the CIDEr drop after ablating the top 200 attributed latents, as curvature balancing strengthens.Intermediate values also decline monotonically: 90.4% at α = 0.25, 85.6% at α = 0.5, and 80.4% at α = 0.75.Source: §I.1 · p. 24
  • 0.994 Spearman correlation on VizWiz at α = 0.5 accompanies an unchanged top-ranked training example for every test example and mean top-8 overlap of 1.00.The balanced and standard representation-gradient influence rankings are compared over 60 evaluation examples.Source: §I.1 · p. 24
  • 0.73 cosine similarity between image- and text-activation mass profiles (0.733 among shared-active latents) indicates that the modalities activate many of the same latent directions.Thus, image-token dominance does not correspond to an independent image direction that can simply be downweighted relative to text.Source: §I.1 · pp. 24–25

I.2 Contrasting Influence With and Without Visual Evidence

  • ρ = 0.543–0.795 for latent influence versus ρ = −0.797–−0.564 for example influence across six settings; image removal also yields Jaccard overlap of 0.021–0.145 and top-8 overlap at or below 0.073.The table defines latent ρ as absolute latent influence, example ρunion over examples active in either condition, and example ρshared over candidates retrieved in both.Source: Table 7 · p. 25, §I.2 · p. 25

I.1 Modality Imbalance

  • 0.999 cosine similarity at α = 0.5 shows that the nominally balanced influence direction is nearly identical to the unrestricted direction.The reported similarities for α = 0, 0.25, 0.75, and 1 are 0.944, 0.988, 0.973, and 0.916, respectively.Source: §I.1 · p. 25

J Additional Qualitative and Hierarchical Attribution Analysis

  • Across VizWiz and GQA, learned latents show clearer semantic associations and more concentrated visual support than base-SAE features.The examples illustrate localized, interpretable features across datasets and task domains.Source: J Additional Qualitative and… · pp. 25–26

J.1 Detailed Analysis of Influential Test Latents

Test-conditioned decompositions show that distinct latents capture complementary objects, interactions, and scene context while remaining grounded in image regions. Influential training examples likewise align with test predictions, through direct answer matches in open-ended tasks and related evidence across clinical labels.

J.1 Detailed Analysis of Influential Test Latents

  • Medicine-image latents separately represent the bottle, medicine semantics, and the hand holding it, localizing object identity, function, and interaction evidence.This gives a richer account of the prediction than a single global visual feature.Source: §J.1 · pp. 26–27
  • Eye-drop-image latents represent eyes, eye drops, and the bottle alongside localized context such as the wooden tabletop.The tabletop is not the primary answer concept, but it is a visible, image-grounded component of the scene.Source: §J.1 · p. 27
  • Across the two examples, semantically related concepts occupy distinct complementary latents whose visual support tracks objects, parts, interactions, and scene context.The decomposition traces both where evidence appears and how multiple pieces jointly support a prediction.Source: §J.1 · pp. 26–27

J.2 Qualitative Analysis of Influential Training Examples

  • For the VizWiz medicine query, top-ranked training examples span medicine semantics, related containers and packaging, and contextual evidence such as the wooden tabletop.The examples provide complementary semantic and visual-context evidence rather than simply repeating the same object.Source: §J.2 · p. 27
  • For VizWiz and GQA, influential training examples match test content directly: screen-related examples align with a screen query, while the top three jeans examples match the queried answer.In the window-identity example, the highlighted answers in the highest-influence examples also say “man.”Source: §J.2 · p. 28
  • CheXpert influential examples can cross diagnostic labels while retaining related clinical evidence, including congestion for edema and overlapping findings for pleural effusion and cardiomegaly.The pleural-effusion examples include both a directly matching case and cases associated with cardiomegaly, atelectasis, and consolidation.Source: §J.2 · p. 28

K Comparison with Pooled Multimodal SAE Baselines

Pooled multimodal SAE baselines can align image and text representations, but their example-level features do not provide the localized, position-resolved evidence needed for grounded influence attribution. MOSAIC retains selective visual support, while inserting pooled dictionaries position-wise does not preserve the VLM computation at the tested layer.

K Comparison with Pooled Multimodal SAE Baselines

  • JSAE and SPARC align image- and text-level representations, but their active visual features span essentially the full image; MOSAIC retains thousands of live features with more selective, localized support.Table 8 defines footprint as the fraction of the image patch grid activated and contiguity relative to size-matched random support; lower footprint, contiguity, and word concentration indicate more selective features.Source: Table 8 · p. 29, K Comparison with Pooled Multimodal… · p. 29
  • When inserted position-wise at the intermediate layer, pooled dictionaries do not preserve the VLM computation; JSAE approaches the mean-broadcast control, while SPARC performs worse.Table 9 reports mean cross-entropy over 60 VizWiz validation examples, with Δ relative to MOSAIC; the supplied passages do not state the cell values.Source: Table 9 · p. 29, K Comparison with Pooled Multimodal… · p. 29
  • Pooled alignment does not define phrase–patch correspondence: the baselines align mean-pooled or stream-level representations, whereas MOSAIC uses position-resolved representations and phrase–region supervision.An example-level shared concept alone cannot identify which patch or phrase carries training-example influence.Source: K Comparison with Pooled Multimodal… · p. 29
  • MOSAIC is designed to preserve the forward computation while providing a shared latent basis resolved at phrase, token, patch, and region levels for grounded influence decomposition.The comparison distinguishes cross-modal alignment from an attribution-compatible multimodal basis.Source: K Comparison with Pooled Multimodal… · p. 30

Related papers

Loading 2610.14190v1…