Multi-Object Hallucination in Vision-Language Models

• Existing benchmarks primarily test object-class presence, which may not establish recognition of every individual referent or freedom from hallucination.Object classes and individual objects can map many-to-many, and text-based yes/no questions differ from simultaneous multi-object recognition.Source: §1 · pp. 1–2
• GPT-4V identifies a prompted whisk beside a knife but hallucinates a fork when asked to recognize both objects together.The authors connect multi-object querying to embodied tasks such as recognizing cooking tools and ingredients, and note that simultaneous queries can save time and resources.Source: §1 · p. 2, Figure 1 · p. 2
• ROPE evaluates concurrent recognition of multiple visually referred objects using an automated protocol with controlled output formatting.Its test subsets account for the distribution of object classes within an image.Source: §1 · p. 2
• LVLMs hallucinate more when focusing on multiple objects than on a single object, and their behavior varies with class distribution, data-specific factors, salience, frequency, and model-intrinsic behavior.The findings suggest possible shortcuts and spurious correlations; the authors propose balanced object distributions, diverse annotations, and enhanced multi-object instructions.Source: §1 · p. 3

• Visual prompting uses image cues, such as marks or visual text, and ROPE applies it to reduce ambiguity when referring to objects, including multiple objects of the same class.Source: §2 · p. 3
• Object-hallucination research includes mitigation methods and benchmarks, but the effect of object-class distribution within an image at test time remains under-explored.Source: §2 · p. 3
• ROPE evaluates multiple object classes and their test-time distribution, refers to objects with visual cues such as marked boxes, and uses automated evaluation rather than human or black-box neural evaluators.Source: §2 · pp. 3–4

3 Recognition-based Object Probing Evaluation

• ROPE evaluates how LVLMs misperceive multiple objects queried concurrently, including by inventing objects or becoming distracted.Source: §3 · p. 4

3.1 Task Setup

• Each ROPE sample pairs an image and recognition instruction with visual prompts for n objects and their correct classes; the example dataset tests five objects from 50 candidate classes.The output format supports automated parsing; default prompts query all five objects together, while student-forcing constrains the format and teacher-forcing supplies ground-truth prior object tokens to limit cumulative error.Source: §3.1 · p. 4

3.2 Dataset Construction

• The authors build the dataset from MSCOCO-Panoptic and ADE20K and divide evaluation by object-class distribution and whether images appeared in the datasets’ training split.The distribution subsets include homogeneous, heterogeneous, adversarial, and in-the-wild samples; the seen/unseen distinction may not be strict.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

4.1 LVLM Baselines

• The baselines span different model scales and tuning types; visual prompts use model-specific mechanisms or overlaid indexed bounding boxes.The evaluation includes mechanistically grounded models, for which the authors also test the default format and report the better-performing format.Source: §4.1 · pp. 5–6, Table 2 · p. 6
• The authors caution that the seen-versus-unseen image distinction may not be strict because web-image overlap and opaque training data complicate the split.They describe the split as their best-practice approach while acknowledging its limits.Source: §4.1 · p. 5

4.2 Main Results and Findings

• Across most models and splits, multi-object queries have lower accuracy than single-object queries, and greater query heterogeneity brings more hallucinations.The exceptions include GPT-4O, MiniCPM-V, and CogVLM-2; teacher-forced homogeneous queries also show unusually high accuracy.Source: §4.2 · p. 7
• On LLaVA’s unseen adversarial split, accuracy for the final, different-class object falls nearly to zero, versus 23.35% when probed individually and 19.16% when queried first.The authors interpret the contrast as evidence that high homogeneous-query performance can arise from textual shortcuts.Source: §4.2 · p. 7, Figure 4 · p. 7
• The multi-object hallucination gap and reliance on shortcuts persist across seen and unseen images, while training on these images does not significantly reduce hallucinations.Most models perform slightly better on seen images, but the overall trends remain consistent across both splits.Source: §4.2 · pp. 7–8

4.3 What May Help and What May Not?

• Larger base LLMs improve single-object results but not consistently unseen multi-object results; the authors suggest greater memorization of seen images may contribute.Some LLaMA-3-based models are more robust to multiple visual prompts but underperform quantitatively due to instruction-following errors.Source: §4.3 · p. 8
• Conversational tuning can reduce multi-object hallucinations, while grounded tuning has not reliably closed the multi-object gap in the tested settings.The authors note that grounded instruction data often consists of short captions or questions featuring one or a few objects; MiniCPM-V shows stronger recognition when same-class objects are probed together.Source: §4.3 · p. 8
• OPERA offers marginal gains for LLaVA-1.5 in default multi-object tasks, but its effectiveness declines with greater heterogeneity and can lower performance.The authors characterize it as beneficial in homogeneous contexts but needing refinement for mixed-object scenarios.Source: §4.3 · p. 8

5.1 Potential Hallucinatory Factors

• The analysis considers input order, query and object homogeneity, token position, object centrality, object salience, semantic salience, and training frequency.It compares factors between hallucinatory and non-hallucinatory objects in LLaVA-13B on the unseen split under student forcing; continuous factors use ridgeline plots and discrete factors use bar charts.Source: §5.1 · pp. 8–9, §5.2 · p. 9
• Model-behavior measures include entropy of the first generated-word token and visual modality contribution, the attention allocated to visual versus textual tokens.The authors treat higher entropy as greater prediction uncertainty and lower visual contribution as potentially indicating less attention to visual cues.Source: §5.1 · p. 9

5.2 When Do LVLMs Experience Multi-Object Hallucinations?

• Hallucinations increase with query or object heterogeneity; token position has minimal impact, while off-center objects show a slight increase.The authors suggest the positional tendency may reflect reporting bias toward foreground objects near image centers.Source: §5.2 · p. 10
• Models more often hallucinate less salient classes and classes infrequent in training, while individual-object salience does not statistically correlate with hallucination.The authors infer that co-occurring same-class objects may matter more than an individual object's presence or salience.Source: §5.2 · p. 10
• Higher token entropy accompanies more hallucination, visual attention remains below 20%, and lower visual attention has a marginal association with hallucination.The authors interpret the low visual-modality contribution as suggesting greater reliance on linguistic context.Source: §5.2 · p. 10

5.3 How Do LVLMs Experience Multi-Object Hallucinations?

• Hallucinated predictions favor classes common in training and candidates listed early in the prompt; semantic salience has little effect on which class is predicted.The authors conclude that spurious correlations may contribute to multi-object hallucinations.Source: §5.3 · p. 10

6 Discussions and Conclusion

• The authors report greater hallucination with multiple objects than with single objects, and increased risk in heterogeneous data and low-certainty scenarios.They suggest probing objects individually in visual prompts may enhance performance.Source: §6 · p. 10
• The conclusion says grounded instruction tuning and scaling the base language model may not be enough to fully address object hallucination.It calls for more balanced object distributions, annotations of objects away from image centers, and greater diversity.Source: §6 · p. 10
• The authors call for instructions requiring multiple visual pointers and complex multi-object reasoning to address realistic multi-object scenes.They frame this as important for enabling LVLMs to recognize and reason about multiple objects.Source: §6 · p. 10

A.1 Reproducibility

• The curation pipeline keeps top-50 “thing” classes, excludes objects below 1% image area, requires at least five valid objects per image, and limits box IoU to 0.1.The pipeline is applied to MSCOCO-Panoptic and ADE20K.Source: §A.1 · p. 16
• The prompt materials cover single- and multi-object queries, with visual pointers and formatted class outputs for the five queried objects.The examples include model-specific prompt templates and a class list for autonomous-driving objects.Source: §A.1 · pp. 16–17, Figure 7 · p. 16
• Experiments used eight A40 and four A100 GPUs for slightly over a week; collecting logits and attention values was the computational bottleneck.Source: §A.1 · p. 17

A.2 Additional Experiments and Results

• Per-object performance is reported in Tables 5 and 6, but this passage gives no cell values or comparisons.Source: §A.2 · p. 17
• The nuScenes case study reports more hallucination errors in the multi-object case than in the single-object case.The authors connect this comparison to the need to detect multiple objects accurately in real-world autonomous-driving scenarios.Source: §A.2 · p. 18
• Figure 8 illustrates single- and multi-object cases in the nuScenes autonomous-driving setting, with each object identified independently.Source: §A.2 · p. 18

B.1 Limitations

• ROPE’s unseen dataset may have been previously exposed, because LVLMs lack transparency.Source: §B.1 · p. 18
• A fixed set of semantic objects may bias evaluation and constrain instruction following and reasoning.Source: §B.1 · p. 18
• Evaluation can be slow because it requires five inferences per image for both student forcing and teacher forcing.Source: §B.1 · p. 18

B.2 Artifacts and licenses

• The authors report licenses for the datasets and models used and limit model use to academic research.Table 4 presents the artifact license information.Source: §B.2 · p. 18, Table 4 · p. 18

B.3 Ethical concerns and risks

• The study uses no human annotators or participants in its interactive experiments.Source: §B.3 · p. 18
• Public datasets and model-created content may introduce biases and sensitive elements.Source: §B.3 · p. 18
• The authors identify fairness-based filtering and metrics as possible directions for future research addressing public-data concerns.Source: §B.3 · p. 18