VCoder: Versatile Vision Encoders for Multimodal Large Language Models

1. Introduction

• Existing MLLMs can handle complex vision-language tasks but often miscount or hallucinate objects, with perception worse in cluttered scenes.The paper frames accurate perception as a foundation for describing and reasoning about visual scenes.Source: §1 · pp. 1–2
• COST uses COCO images and off-the-shelf perception-model predictions to create question-answer pairs about foreground and background objects.The dataset addresses the stated scarcity of image-text data focused on fundamental object perception.Source: §1 · p. 2
• VCoder adds segmentation or depth maps as control inputs through additional vision encoders that project their information into the LLM space.The paper proposes these perception modalities as an approach to improving object perception.Source: §1 · p. 3
• The proposed count score uses one-to-one matching of object words, while hallucination and depth scores measure extra objects and object-order prediction.The paper introduces these metrics to quantify object perception, especially counting ability.Source: §1 · p. 3
• VCoder-adapted LLaVA-1.5 outperforms baseline MLLMs on all metrics when validated on COST.The metrics cover object perception, including counting, hallucination, and depth-based order prediction.Source: §1 · p. 3

2.1. Visual Perception

• The paper focuses on object-level perception, especially predicting object names, counts, and order in images using MLLMs.Visual perception also includes dense tasks such as segmentation and depth estimation, and sparse tasks such as detection and pose estimation.Source: §2.1 · p. 3

2.2. Visual Understanding with LLMs

• Prior MLLM approaches project vision features into an LLM embedding space and instruction-tune on vision-language dialogue; LLaVA-1.5 is the paper’s baseline.The related work describes different vision encoders, projectors, and fine-tuning approaches, including LLaVA-1.5’s MLP projector.Source: §2.2 · p. 3

2.3. Perception Hallucination in MLLMs

• Unlike prior hallucination benchmarks centered on VQA or object presence and caption words, this work evaluates object-level hallucination using object words and counts.COST uses segmentation outputs for object identification and adds depth maps for object-order perception.Source: §2.3 · pp. 3–4, Figure 3 · p. 4

3. Object Identification with MLLMs

• MLLMs can describe a scene yet miscount its entities; the example GPT-4V answer counts ten people where the image contains fourteen.The section characterizes this gap as MLLMs failing at simple visual perception while succeeding at complex visual reasoning.Source: §3 · p. 4

3.1. COST to Identify Objects with MLLMs

• COST pairs COCO images with OneFormer segmentation outputs, object names and counts, and GPT-4-generated questions for semantic, instance, and panoptic identification.The training visual component uses 280k images from COCO train2017, test2017, and unlabeled2017; validation uses 5k val2017 images.Source: §3.1 · p. 5
• For object-order perception, COST uses DINOv2 DPT depth maps and OneFormer panoptic masks to derive object order and formulate question-answer examples.Repeated objects of the same class receive count suffixes to preserve their relative ordering.Source: §3.1 · p. 5

3.2. VCoder for Multimodal LLMs

• VCoder adds segmentation-map control to an MLLM through a pretrained ViT and two-layer MLP, while keeping the base model’s parameters fixed during training.The adapter is added to LLaVA-1.5; its Image enCoder, MLP, and LLM are frozen to preserve original reasoning performance.Source: Figure 4 · p. 5, §3.2 · p. 6

3.3. Evaluating MLLMs for Object Identification

• Count Score measures the percentage of correct predicted object counts, while Hallucination Score measures extra predicted object counts absent from ground truth.The scores use one-to-one matching of object words and counts between the ground-truth sentence and MLLM response.Source: §3.3 · p. 6
• Evaluation manually maps COCO categories to synonyms, replacing terms such as “man” or “child” with “person” before one-to-one matching.This normalization addresses the evaluation’s reliance on exact object-word matching.Source: §3.3 · p. 6

4.2. Main Results

• VCoder-adapted LLaVA-1.5 performs best on all three object-identification tasks and outperforms GPT-4V on the reported panoptic comparison.Existing MLLMs perform poorly on COST, while segmentation control performs considerably better than RGB control.Source: Table 1 · p. 7, §4.2 · p. 8

4.1. Implementation Details

• Experiments use VCoder-adapted LLaVA-1.5, projecting features into the LLM embedding space with modality-specific two-layer MLPs; 336×336 visual inputs yield 576 tokens, while instruction-tuned weights are frozen and only VCoder MLPs tuned.Training runs for two epochs on COST with batch size 256 and learning rate 1e−3. Evaluation separately samples questions from semantic, instance, and panoptic task buckets on the COST validation set.Source: §4 · p. 7, §4.1 · p. 7

4.2. Main Results

• The comparisons include four off-the-shelf MLLMs and three COST-trained LLaVA-1.5 variants: COST IT, Soft-Prompted, and ImCoder.ImCoder uses RGB as its control input instead of a segmentation map.Source: §4.2 · pp. 7–8
• VCoder LLaVA-1.5 considerably outperforms base LLaVA-1.5 on object order perception.The paper attributes this comparison to control inputs and COST training.Source: Table 2 · p. 8

5. Object Order Perception with MLLMs

• An inaccurate segmentation map used as the control input can lead VCoder to return a wrong response.Source: Figure 6 · p. 8
• The object-order task uses both depth and segmentation maps as controls, while object identification uses segmentation alone.The approach treats object order prediction as implicitly involving object identification.Source: §5 · p. 8
• Training combines COST object-identification and object-order data with about 200k image-conversation pairs and corresponding OneFormer segmentation maps.The VCoder is trained for one epoch.Source: §5 · p. 8
• VCoder-DS LLaVA-1.5 significantly outperforms base LLaVA-1.5 on COST validation for object order perception.The depth score measures the absolute difference between object positions in the ground truth and prediction.Source: §5 · p. 8

6. Limitations

• COST is built with OneFormer, whose closed-set COCO training limits the object categories it can perceive.The authors call for benchmarks covering more classes and varying granularity than COCO.Source: §6 · pp. 8–9
• Count, hallucination, and depth scores rely on one-to-one word matching and manually defined mappings between synonyms.The authors identify avoiding manual synonym mapping as a direction for future work.Source: §6 · p. 9
• Inaccurate segmentation maps can cause VCoder to fail, motivating work to reduce its overdependence on control inputs.Source: §6 · pp. 8–9

7. Conclusion

• The authors report that VCoder-adapted LLaVA-1.5 improves object perception while retaining reasoning performance.Source: §7 · p. 9
• COST is proposed for training and evaluating MLLMs on object-level perception, alongside count, hallucination, and depth scores.Source: §7 · p. 9
• VCoder projects perception control inputs into the LLM embedding space and can be extended to different modalities depending on the task.Source: §7 · p. 9
• The authors hope to encourage perception-focused MLLM datasets and vision systems equally capable at perception and reasoning.Source: §7 · p. 9

• The appendix examines segmentation-map quality, explains object-order ground truth and depth-score computation, and analyzes per-image COST object counts.Source: Appendix · p. 13

A. Control Through Segmentation Map

• Performance drops considerably with Mask R-CNN and Panoptic-FPN maps, while the drop is much lower with the relatively newer, better Mask2Former maps.The comparison is presented for instance and panoptic object identification.Source: A. Control Through Segmentation Map · p. 13

B.1. Obtaining Ground Truth

• The object-order procedure uses segmentation and depth maps to establish ground truth, while the appendix also introduces the depth-score computation.Source: B. Object Order Perception · p. 13

B. Object Order Perception

• Ground-truth object order is derived by sorting each object's farthest-point depth values from closest to farthest.Binary masks select object regions in DINOv2 DPT depth maps; objects in the same category receive numbers representing their relative order.Source: §B.1 · p. 13

B.2. Depth Score

• The depth score uses absolute differences between predicted and ground-truth positions of objects belonging to the same category.Source: §B.2 · p. 13
• Unmatched objects receive position value 100, and the resulting score is averaged over all images.Source: §B.2 · p. 13

C. Object Counts in COST Dataset

• The observed long tail motivates more scaled object-level perception dataset collection for training MLLMs to count in cluttered scenes with many objects.The stated goal is counting without extra pre-processing.Source: C. Object Counts in COST Dataset · p. 13
• COST contains no images with more than 60 objects, while its per-image object-count distribution has a long tail beyond 25.Source: C. Object Counts in COST Dataset · p. 13, Figure III · p. 14