Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models

1 Introduction

• VALUE probes multimodal fusion, modality importance, cross-modal, image-to-image, and text-to-text interactions in LXMERT and UNITER.The paper selects LXMERT as its two-stream model and UNITER as its single-stream model to investigate both architectures.Source: §1 · pp. 2–3, Figure 1 · p. 2
• Both single- and two-stream models capture cross-modal alignment, visual relations, and linguistic knowledge, especially the single-stream model; predictions tend to depend more on text than images.The modality finding comes from tracing the [CLS] token’s attention; the interaction findings use Visual Coreference Resolution, Visual Relation Detection, and SentEval.Source: §1 · p. 3
• As layers deepen, image and text representations become more intertwined in single-stream models, whereas two-stream models show the opposite trend.The fusion result is based on clustering image and text representations.Source: §1 · p. 3
• The paper presents VALUE as the first thorough analysis of latent knowledge in pre-trained vision-and-language models from multiple perspectives.Its stated aim is to identify ingredients behind these models’ empirical success.Source: §1 · p. 3

2 Related Work

• Image-text pretraining spans two-stream and single-stream [21] [6] models, unified captioning and VQA, multitask and adversarial methods [10], and video-language approaches [29] [20].Source: §2 · p. 3
• Prior BERT analyses examined linguistic structure, attention-head importance, syntax and semantics, factual and commonsense knowledge, and question-answering reasoning [15] [31] [7] [12].Source: §2 · pp. 3–4
• Systematic analysis of pretrained vision-and-language models remained lacking; this work proposes visual coreference and visual relation probing tasks to investigate their multimodal mechanisms.The authors present these probes as potentially informative for other evaluation tasks as well.Source: §2 · p. 4

3.1 Deep to Profound: Deeper Layers Lead to More Intertwined Multimodal Fusion

• UNITER-base representations fuse progressively across layers, whereas LXMERT representations become more distinct with depth except at the final layer.The study attributes the contrast possibly to UNITER sharing parameters across modalities while LXMERT uses separate parameters for independently modeling image and text.Source: §3.1.1 · p. 6, Table 1 · p. 6

3.2 Who Pulls More Strings: Textual Modality Is More Dominant than Image

• Textual Modality Importance scores exceed visual scores, especially in intermediate UNITER-base layers, indicating greater reliance on text during inference.The textual-MI heatmap is denser than the visual-MI heatmap across the 144 attention heads.Source: §3.2.2 · p. 7, Figure 2 · p. 7

3.3.1 Probing Image-to-text Head

• Only 15% of UNITER-base heads have more than 0.5 probability of attending mostly from image to text, while one head reaches 0.92 probability.This pattern indicates that a subset of heads learns cross-modal interaction and that single-stream attention can develop a two-stream-like division of labor.Source: §3.3.1 · p. 8, Figure 3 · p. 8
• Attention traces encode linked region–phrase coreference, with heads (9-3), (9-12), and (3-1) capturing richer knowledge than other heads, especially from visual tokens to text.Some single-stream heads encode coreference in both directions, while text-to-visual traces capture limited coreference knowledge.Source: §3.3.1 · pp. 9–10, Table 2 · p. 9

3.3.3 Probing Combinations of Heads

• Neither model performs well on Visual Coref Detection, while both embedding probers exceed 94% accuracy on Visual Coref Classification.The authors caution that text embeddings may leak coreference information, so the strong classification accuracy does not necessarily demonstrate multimodal knowledge.Source: Classifier Input VCD (SS) VCD (TS) VCC… · p. 11, §3.4 · p. 11

3.4 Secret Liaison Revealed: Cross-Modality Fusion Registers Visual Relations

• Specific attention heads encode visual relations more strongly for related region pairs than random pairs, with head 10-1 strongest in the single-stream model and head 7-4 in the two-stream model.The reported individual-head analysis covers seven of the 30 relations.Source: §3.4 · p. 12, Table 4 · p. 12
• Related annotations support visual-relation knowledge in both models, but mismatched captions degrade it; the two-stream model suffers a large post-Layer-5 drop in classification, unlike the single-stream model.The authors attribute this difference to two-stream visual representations being required to attend to text, whereas single-stream representations can selectively attend to it.Source: §3.4 · p. 12, Classifier Input VRI (SS) VRC (SS) VRI… · p. 13, Figure 4 · p. 13

3.5 No Lost in Translation: Pre-trained V+L Models Encode Rich Linguistic Knowledge

• The linguistic-knowledge analysis probes nine SentEval tasks using contextualized word representations from the pre-trained models.The supplied passage describes the probing setup but does not report its results.Source: §3.5 · p. 13
• Across linguistic probing tasks (Table 5), pre-trained V+L models generally trail BERT-base, and the single-stream model outperforms the two-stream model on every task.Results use each model’s best score across layers.Source: Table 5 · p. 14, §3.5 · p. 14
• For two-stream models, evaluation covers only the first nine self-attention layers because the final five cross-attention layers require visual input absent from these linguistic benchmarks.The evaluated layers perform self-attention over pure text input.Source: §3.5 · p. 14
• The authors suggest BERT-base initialization may explain the advantage: UNITER initializes its language encoder from BERT-base, whereas LXMERT does not.They note that BERT initialization may help models acquire linguistic knowledge useful for complex text-based reasoning.Source: §3.5 · p. 14
• To measure learning gains, the authors ran the experiments from Sections 3.1–3.5 with random-weight untrained baselines and provided those results in the Appendix.Source: §3.5 · p. 14

• In single-stream models, deeper layers produce more intertwined multimodal fusion, while two-stream models show the opposite trend.Source: §4 · p. 14
• Text plays a more important role than images in final decisions across both single- and two-stream models.Source: §4 · p. 14
• Single-stream models develop a subset of heads focused on cross-modal interaction and alignment, whereas two-stream models enforce these through their design.Source: §4 · p. 14
• Both single- and two-stream pre-trained models inherently register visual relations.Source: §4 · p. 14
• Both models naturally encode rich linguistic knowledge despite being designed for multimodal pre-training.Source: §4 · p. 14
• The authors plan to explore model compression by pruning attention heads based on their analysis and observations.Source: §4 · p. 14

A Details on Pre-trained V+L Models

• Figure 5 compares UNITER-base [6] as a single-stream model with LXMERT as a two-stream model.Source: Figure 5 · p. 17, A Details on Pre-trained V+L Models · p. 17
• UNITER-base [6] matches BERT-base, with 12 self-attention layers, 12 heads per layer, and 768-dimensional hidden representations.Source: A Details on Pre-trained V+L Models · p. 17
• LXMERT uses intra-modal attention before cross-attention, with 9 text, 5 image, and 5 cross-modal layers, each 768-dimensional.Each cross-modal layer has one cross-attention layer followed by two self-attention layers for each modality.Source: A Details on Pre-trained V+L Models · p. 17

Multimodal Fusion Probe

• Untrained SS has NMI 0.99 at every output layer, indicating complete modality separation, whereas untrained TS has NMI 0.56 because its cross-modality encoder forces fusion.Source: B Results on Untrained Baselines · p. 17

Modality Importance Probe

• For untrained models, [CLS] attention averages 0.66 on image and 0.28 on text; sentences usually contain fewer tokens than visual inputs.Source: B Results on Untrained Baselines · p. 17

Visual Coreference and Relation Probe

• VCD and VRI untrained SS and TS baselines match random guess; VRC exceeds baseline by around 10% in both, while VCC is 17% above baseline for SS and worse for TS.The authors suggest hard-designed multimodal fusion may dilute direct coreference relationships between image and text tokens after training.Source: Table 6 · p. 18, B Results on Untrained Baselines · p. 18
• Heads selected for a specific image-text coreference relation assign higher attention to that paired relation than to other unpaired pairs.Source: Table 7 · p. 18, B Results on Untrained Baselines · p. 18

C Additional Guidelines for Future Model Design

• Initializing vision-and-language models with BERT weights should help strengthen their language-understanding capability.Source: C Additional Guidelines for Future… · p. 18
• Probing tasks offer a convenient way to quickly test intermediate pretraining checkpoints because finetuning is time-consuming and downstream evaluation is otherwise unclear.The authors note that it remains unclear how to measure a pretrained model without evaluating it on downstream tasks.Source: C Additional Guidelines for Future… · p. 18
• Adding explicit supervision to probing tasks during training may produce models that are more interpretable and robust.Source: C Additional Guidelines for Future… · p. 18

D Details on Linguistic Probe

• The linguistic probe uses nine SentEval [8] tasks, including sentence length (SentLen), syntax-tree depth, token inversions (BShift), and top-constituent sequences (TopConst).Source: D Details on Linguistic Probe · p. 19
• Its semantic tasks test tense (Tense), subject/object number (SubjNum/ObjNum), noun or verb replacement (SOMO), and coordinating-conjunction inversion (CoordInv).Source: D Details on Linguistic Probe · p. 19

E Additional Results

• The section presents additional results on multimodal fusion, visual coreference resolution, visual relation detection, and linguistic probing.Source: E Additional Results · p. 19

E.1 An t-SNE Visualization of Multimodal Fusion Degree

• UNITER’s visual and textual tokens become more intertwined as layers deepen, as shown in a t-SNE visualization of one image-text pair.Figure 6 represents visual tokens as yellow dots and textual tokens as blue dots.Source: §E.1 · p. 19, Figure 6 · p. 19

E.2 Visual Coreference Resolution

• Visual coreference results are reported using embeddings from Layers 1, 5, and 12; a complete result set is referenced in Table 8.The supplied passage does not provide the result values or comparisons.Source: §E.2 · p. 19

Classifier Input VCD (SS) VCD (TS) VCC (SS) VCC (TS)

• Attention probers work well for VCC but not VCD, which the authors attribute to VCC distinguishing relation-specific attention patterns that VCD training examples dilute.VCC is trained with positive examples from VCD, making it harder to distinguish attention across different coreference relations in VCD.Source: Classifier Input VCD (SS) VCD (TS) VCC… · p. 20

E.3 Visual Relation Detection

• The layer-wise embedding-prober results for Visual Relation Classification are visualized in Figure 4(b), with detailed values in Table 9.Source: §E.3 · p. 20
• For Visual Relation Identification, layer-wise embedding-prober results are visualized in Figure 4(c), with detailed values in Table 10.Source: §E.3 · p. 20

E.4 Linguistic Probing

• Linguistic probing evaluates every pre-trained-model layer and reports the best result in Table 5, with all-layer details in Table 11.Source: §E.4 · p. 20

E.5 Visualization of Attention Maps

• For visual coreference, only a subset of UNITER’s 144 V→T attention heads is significant across relations; LXMERT’s maps visualize coreference across all heads.The LXMERT-base visualization covers five layers with 12 heads per layer.Source: §E.5 · p. 20, Figure 7 · p. 23, Figure 8 · p. 23
• For visual relation detection, UNITER’s attention maps for spatial relations “on” and “at” are similar to each other.The visualization shows maximum attention between visually related tokens across 144 heads.Source: §E.5 · p. 20, Figure 9 · p. 24
• For visual relation detection, LXMERT’s maps show maximum attention between visually related tokens across 10 layers, with cross-self-attention in layers 6–10.The visualization includes 12 heads per layer.Source: §E.5 · p. 20, Figure 10 · p. 24