Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models

1 Introduction

• The study probes representations with simple prediction heads across object names, attributes, detection, instance segmentation, and object-part segmentation.The evaluated models include OFA, FLAVA, CLIP, MAE, and MOCOv3.Source: §1 · p. 1
• VL models outperform vision-only models on label prediction, while vision-only models are stronger on dense prediction tasks requiring localized information.The comparison probes semantics and localization across five tasks.Source: §1 · p. 1, Figure 1 · p. 2
• VL representations encode richer semantic information, but do not enrich the localized information required by spatial-aware tasks.This pattern is supported by detailed analyses of segmentation and attribute prediction.Source: §1 · p. 1
• The probing results are intended as an empirical guide for choosing pretrained models for different downstream tasks.The authors also position the study as a benchmark for publicly released pretrained models.Source: §1 · p. 2

Vision-and-language pretraining

• Vision-and-language pretraining has shown strong performance on multimodal tasks and potential on single-modal vision tasks.Representative methods include CLIP, ALIGN, OFA, Florence, FLAVA, Unified-IO, CoCa, and SimVLM.Source: §2 · p. 2

Vision and language benefit each other

• Earlier research examined how vision improves language understanding, grounding, commonsense knowledge, and reporting-bias mitigation.The cited works study benefits flowing from visual information into language models.Source: §2 · p. 2

Probing

• Probing has been widely used to interpret linguistic and multimodal representations across syntactic, semantic, grounding, retrieval, and commonsense properties.These studies compare what different models encode rather than relying only on downstream performance.Source: §2 · pp. 2–3
• The paper identifies visual probing as an underexplored complement to fine-tuning and presents a comparison of vision-and-language and vision-only models.The authors describe this as the first such comparison using probing.Source: §2 · p. 3

3 Method

• Pretrained visual encoders are frozen while simple task-specific heads predict labels or dense outputs from extracted features.For an image I ∈R3×w×h, the encoder produces f ∈R^C×W×H and only head P is trained.Source: §3 · p. 3

3.1 Probing tasks and datasets

• The five probing tasks comprise object-name prediction, attribute prediction, object detection, instance segmentation, and object-part semantic segmentation.Object names and attributes emphasize semantic labels, whereas the remaining tasks rely heavily on spatial information.Source: §3.1 · p. 3
• The datasets cover Visual Genome, VAW, MSCOCO, and PartImageNet, with task-specific heads and metrics for classification, detection, and segmentation.PartImageNet adds object-part masks for finer-grained localization evaluation, while MSCOCO supplies boxes and masks for 80 categories.Source: §3.1 · pp. 3–4, Table 1 · p. 4

3.2 Evaluated models

• The comparison evaluates three vision-and-language models—CLIP, OFA, and FLAVA—and two vision-only models—MAE and MOCOv3.The models are selected as representative, popular systems with publicly available pretrained weights and code.Source: §3.2 · p. 4
• Model-specific visual features are extracted from each pretrained architecture, including CLIP image-encoder embeddings, OFA transformer-encoder outputs, and FLAVA visual-encoder outputs.MAE and MOCOv3 provide vision-only transformer representations based on masked-image modeling and contrastive learning, respectively.Source: §3.2 · pp. 4–5

3.3 Comparison settings

• All models use 224*224 inputs, while model sizes, patch sizes, and feature dimensions are aligned as closely as available checkpoints permit.The fixed resolution prioritizes fair comparison over the best standalone dense-task performance; probing is sensitive to input and patch sizes.Source: §3.3 · p. 5

4.2 Probing results

• VL models consistently outperform vision-only models on object-name and attribute prediction, with over 51% versus under 50% accuracy for object names and above 61% versus below 55% mAP for attributes.These results indicate that VL representations encode richer semantic information that can be decoded with a simple linear layer.Source: §4.2 · p. 6
• MAE performs best on all three dense prediction tasks, while instance segmentation separates it from other models by more than 2.7% mAP.OFA is close to MAE for COCO detection, at 25.0 versus 25.3 mAP, but MAE’s advantage is clearer for instance segmentation.Source: §4.2 · p. 6, Table 2 · p. 6
• Vision-only models have consistently smaller performance drops from COCO detection to instance segmentation, indicating more localized representations.The two tasks use the same metrics and dataset, differing in whether overlap is measured with bounding-box or pixel-wise IOU.Source: §4.2 · p. 6
• Segmentation diagnostics show VL models are better at semantic label prediction, whereas vision-only models are better at localization and object-boundary prediction.On MSCOCO, VL models exceed 60 F1 while MAE and MOCOv3 remain below 50; the diagnostic metrics separately evaluate semantics and foreground-background localization.Source: Table 3 · p. 6, §4.2 · p. 7
• VL advantages in attribute prediction are largest for abstract categories such as action, while vision-only models are relatively better on visually grounded attributes such as texture.The VAW analysis separates action, texture, shape, size, color, material, state, and other attributes.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 More analysis

• Contrastive models show relatively stronger localization on single-object PartImageNet images than on multi-object MSCOCO images.CLIP exceeds OFA and FLAVA by more than 6% mIOU on PartImageNet but underperforms them on MSCOCO; MOCOv3’s gap to MAE is 2.2 versus 5.1 across the datasets.Source: §4.3 · pp. 7–8
• Model size has a smaller effect than model type or input image size: larger MAE models improve probing by 1% to 2%, whereas larger OFA models usually decline under the common 224*224 probing input.OFA’s larger model was pretrained at 480*480 versus 384*384 for the base model, creating a larger pretraining-to-probing resolution gap.Source: §4.3 · p. 8, Table 4 · p. 8
• Downstream finetuning usually hurts probing performance, but VQA finetuning causes a smaller decrease than ImageNet finetuning except for attribute prediction.The authors attribute the difference to VQA retaining more fine-grained information about objects and attributes.Source: §4.3 · pp. 8–9, Table 5 · p. 9

• VL models are stronger in label prediction tasks, while vision-only models perform better in dense prediction tasks.Source: §5 · p. 9

• The evaluation covers only publicly accessible checkpoints that could be aligned in model size, patch size, and related properties.The authors could not retrain models because of limited computational resources.Source: §6 · p. 9
• The compared models remain misaligned in training data, architecture, objectives, and hyperparameters, limiting the comparison’s control.Aligning these components would require substantial GPU resources and training effort.Source: §6 · p. 9

• Standard deviations are reported from three repeated experiment runs to indicate the significance of the probing results.Source: A Appendix · p. 12, Table 6 · p. 12
• Detailed numerical attribute-prediction results corresponding to Figure 3 are provided in Table 7.Source: A Appendix · p. 12, Table 7 · p. 13
• Table 8 compares the evaluated models by feature size, architecture, training data, and training objectives.Source: A Appendix · p. 12, Table 8 · p. 13