Source-linked AI summary
GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMS
Saim Rehman, Muhammad Shafique
TL;DR
It is unclear whether quantization that preserves multimodal utility also preserves visual grounding, and whether lower precision yields lower latency. GHOST-Q compares FP16, INT8, and NF4 VLMs across three architectures using matched examples, paired reliability analysis, deployment-efficiency measurements, and open-ended generation-budget sensitivity. Preserved aggregate utility does not guarantee preserved grounding or deployment efficiency; 10 of 36 paired comparisons remain significant after FDR correction, nine on hallucination-sensitive conditions.
Problem
It is unclear whether quantization that preserves multimodal utility also preserves visual grounding, and whether lower precision yields lower latency.
Source: §1 · p. 1Method
GHOST-Q compares FP16, INT8, and NF4 VLMs across three architectures using matched examples, paired reliability analysis, deployment-efficiency measurements, and open-ended generation-budget sensitivity.
Source: §1 · p. 1Results
Preserved aggregate utility does not guarantee preserved grounding or deployment efficiency; 10 of 36 paired comparisons remain significant after FDR correction, nine on hallucination-sensitive conditions.
Source: §5.2 · p. 3, §6 · p. 4Takeaways & Limitations
The results support evaluating quantized VLMs jointly for utility, paired grounding reliability, generation conditions, and realized deployment behavior.
Source: §6 · p. 4The study is limited to three model families, bitsandbytes weight quantization, and batch-one A100 inference.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Post-training quantization of vision--language models (VLMs) is typically assessed through aggregate task accuracy and memory savings, but preserving a headline score does not guarantee preservation of visual grounding behavior. We present GHOST-Q, a cross-precision controlled evaluation of three 8B VLM families under FP16, INT8, and NF4 across utility and hallucination-sensitive benchmarks. Rather than comparing only aggregate accuracy, we pair FP16 and quantized predictions item by-item to quantify how compression redistributes grounding successes and failures. Five of six quantized variants preserve MMStar accuracy within $\pm2$ percentage points, yet 10 of 36 paired effects remain significant after false-discovery-rate correction, nine on hallucination-sensitive conditions. Same-device A100 profiling further demonstrates that substantial memory reduction does not necessarily mean lower inference latency. Finally, an open-ended AMBER audit reveals strong generation budget censoring whose severity varies by architecture and precision. These results show that quantized VLMs should be evaluated jointly for aggregate utility, grounding reliability, generation behavior, and realized deployment efficiency.
1. INTRODUCTION
The paper asks whether quantization preserves visual grounding when it preserves multimodal utility, and whether lower precision translates into lower latency. GHOST-Q evaluates matched precision variants and adds item-level reliability, deployment, and open-ended generation analyses.
Motivating case study
- Idefics3-NF4 changes MMStar accuracy by only +0.40 pp, yet POPE accuracy falls by 1.24 pp, with 257 FP16-correct predictions becoming wrong versus 145 FP16 errors corrected.The paired effect remains significant after BH adjustment (q = 3.0 × 10−7).Source: §1 · p. 1
Contributions
- GHOST-Q compares FP16, INT8, and NF4 on matched examples, preprocessing, prompts, and decoding conditions across three VLM architectures and multiple benchmarks.Source: §1 · p. 1
- Its item-level paired reliability analysis distinguishes harmful and beneficial precision-induced flips using bootstrap intervals, exact McNemar tests, FDR correction, and matched-pair effect sizes.Source: §1 · p. 1
- The evaluation jointly measures behavioral reliability and realized deployment efficiency on the same A100 platform, testing when memory compression is decoupled from inference latency.Source: §1 · p. 1
- An open-ended AMBER audit tests generation-budget sensitivity, finding that fixed decoding budgets can cause architecture- and precision-dependent censoring that confounds naive hallucination comparisons.Source: §1 · p. 1
2. BACKGROUND
Prior quantization work develops methods for reducing transformer and VLM memory, while grounding research evaluates hallucinations through dedicated benchmarks. GHOST-Q focuses on whether compression changes grounding-sensitive decisions despite nearly preserved aggregate utility.
Quantization
- Quantization methods include mixed-precision 8-bit multiplication, second-order one-shot low-bit quantization, W8A8 PTQ, activation-salient channel protection, and NF4 with double quantization.VLM-specific approaches model cross-layer dependencies, modality-specific token sensitivity, token importance, or component sensitivity.Source: §2 · p. 2
Grounding and hallucination
- POPE evaluates object hallucination, AMBER tests existence, attribute, and relation hallucination, and HallusionBench probes image-context reasoning with language hallucinations and visual illusions.GHOST-Q asks whether compression changes grounding decisions even when aggregate utility is nearly preserved.Source: §2 · p. 2
3. METHODOLOGY
GHOST-Q compares each quantized model with its own FP16 reference under matched conditions. It combines item-level flip analysis with an open-ended generation audit to examine precision-sensitive decisions and decoding behavior.
3.1. Controlled Cross-Precision Evaluation
- For each architecture, precision, dataset, and example, candidate responses are deterministically scored from model logits, and accuracy differences are measured against the model's FP16 reference.Comparisons use the same examples, preprocessing, and candidate sets; the ±2 pp MMStar threshold is a reporting tolerance, not a statistical equivalence test.Source: §3.1 · p. 2
3.2. Paired Grounding-Shift Analysis
- Item-by-item analysis counts harmful correct-to-wrong and beneficial wrong-to-correct flips, estimates confidence intervals with 10,000 paired bootstrap resamples, and tests discordant pairs with exact McNemar tests.Benjamini–Hochberg correction covers all 36 comparisons; matched odds ratios summarize flip direction, with values above 1 indicating more beneficial flips and below 1 the reverse.Source: §3.2 · p. 2
3.3. Open-Ended Generation Audit
- The open-ended AMBER audit uses identical prompts, greedy decoding, and a shared token budget, recording natural termination versus reaching the generation ceiling.A higher-budget FP16 stopping-length probe distinguishes natural stopping from budget-induced truncation.Source: §3.3 · p. 2
4. EXPERIMENTAL SETUP
The experiments cover three 8B VLM families, utility and grounding-sensitive benchmarks, and open-ended AMBER generation. Hardware measurements profile all configurations sequentially on one A100.
4.1. Models and Quantization
- Qwen3-VL-8B, InternVL3-8B, and Idefics3-8B are evaluated in FP16, bitsandbytes LLM.int8(), and NF4; 92.35–93.38% of logical linear weights are quantized, while the final LM head remains FP16.NF4 uses FP16 compute and double quantization; INT8 uses the mixed-precision LLM.int8() path, and neither loader requires calibration data.Source: §4.1 · p. 2
4.2. Benchmarks
- Forced-choice evaluation spans MMStar, POPE, three AMBER subsets, and HallusionBench, totaling 25,667 predictions per configuration and 231,003 overall.Open-ended evaluation uses 1,004 AMBER images, the official prompt, greedy decoding, and a common 256-token ceiling; Qwen3-VL uses a fixed 1-megapixel image budget across precisions.Source: §4.2 · p. 2
4.3. Hardware Profiling
- All nine configurations are profiled sequentially on the same A100-SXM4-80GB GPU, with three measured repeats over the same 200 POPE inputs after 10 warm-up examples.Measurements include runtime footprint, peak allocated memory, median and p95 end-to-end latency, throughput, and repeat-wise variability; power is run-level characterization, not per-inference energy.Source: §4.3 · pp. 2–3
5. RESULTS
Across four results subsections, GHOST-Q finds that near-preserved aggregate accuracy can mask paired grounding shifts, while memory savings do not ensure lower latency on the tested A100 stack. Open-ended AMBER evaluation also shows substantial generation-budget censoring that varies by architecture and precision.
5.1. Aggregate Utility is Mostly Preserved
- Five of six quantized variants stay within the ±2 pp MMStar tolerance; InternVL3-NF4 is the exception at −2.07 pp versus FP16.Table 1 summarizes utility and grounding-sensitive accuracy.Source: §5.1 · p. 3
5.2. Preserved Averages Hide Paired Grounding Shifts
- 10 of 36 paired comparisons remain significant after FDR correction, nine on hallucination-sensitive conditions, despite grounding macro changes of at most 0.43 pp.Effects are bidirectional, with matched odds ratios ranging from 13.0 for Qwen3 INT8 on AMBER existence to 0.52 for Qwen3 NF4 on POPE.Source: Table 2 · p. 3, §5.2 · p. 3, Figure 2 · p. 4
5.3. Memory Compression Does Not Imply A100 Speedup
- INT8 reduces model footprint by 42.9–43.8% and NF4 by 64.3–65.7%, but median end-to-end latency rises over FP16 on the tested A100 stack.INT8 latency is 43.5–280.4% higher and NF4 latency is 2.2–69.6% higher; NF4 has lower footprint and median latency than INT8 for all three architectures.Source: §5.3 · pp. 3–4, Table 3 · p. 4
5.4. Open-ended Generation Reveals Budget Sensitivity
- 7,977 of 9,036 AMBER responses (88.28%) hit the common 256-token ceiling, and censoring varies across architectures and precision settings.A separate 1,024-token FP16 diagnostic supports treating the 256-token experiment as a budget-sensitivity audit, not an uncensored generative-hallucination estimate.Source: Table 4 · p. 4, §5.4 · p. 4, Figure 3 · p. 4
6. DISCUSSION AND CONCLUSION
Across three 8B VLM families, GHOST-Q finds that preserved aggregate utility does not guarantee preserved grounding behavior or realized deployment efficiency. Its results also show precision-dependent generation-budget censoring and are scoped to three model families, bitsandbytes weight quantization, and batch-one A100 inference.
6. DISCUSSION AND CONCLUSION
- Quantization can redistribute item-level errors, and substantial memory savings need not reduce latency on the tested stack.The authors conclude that quantized VLMs should be evaluated jointly for utility, paired grounding reliability, generation conditions, and realized deployment behavior.Source: §6 · p. 4
Generative AI Use Disclosure
The authors disclose using ChatGPT and Grammarly for language and visual refinement, with review and editing of the resulting content.
Generative AI Use Disclosure
- The authors used ChatGPT and Grammarly for language editing, text refinement, and methodology-figure refinement, then reviewed and edited the content.They take full responsibility for the publication’s content.Source: Generative AI Use Disclosure · p. 5