PaperlayerView exact revision on arXiv

Source-linked AI summary

BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering

Sourajit Saha, Shubhashis Roy Dipta, Shaswati Saha, Nobin Sarwar, Yuxuan Jiang

arXiv:2609.06922v1cs.CV

TL;DR

VLM quantization can leave VQA answers unchanged while shifting the scores behind them, a behavior benchmark accuracy alone does not measure. BEFORE THE FLIP compares compression-induced shifts with shifts from image-token replacement, probes individual weight groups, and tests question-specific precision selection. W4 shifts LLaVA scores farther toward the replacement output than W8, but per-question selection does not reliably outperform shuffled controls.

  • Problem

    Benchmark scores predominantly report answers, leaving principled evaluation of score changes in quantized VLMs at deployment lacking.

    Source: §1 · p. 2

  • Method

    BEFORE THE FLIP compares compression-induced answer-score changes with changes from replacing image tokens, then tests individual-group precision increases and per-question selection against shuffled controls.

    Source: §1 · p. 2

  • Results

    Mean PRS rises from 0.0094 under W8 to 0.0760 under W4 across 8,277 questions, while only 265 of 9,000 LLaVA answers change at W4.

    Source: §4.2.1 · pp. 5–6

  • Takeaways & Limitations

    Quantization can shift scores before answers change, but the tested results do not support reliable precision adjustment for each question.

    Source: §5 · p. 8

    PRS is limited to binary answers and a replacement favoring NO, and mean PRS lacks image-level confidence intervals.

    Source: §6 · p. 8

Abstract

from arXiv · show

Quantization makes vision language models (VLMs) cheaper to store and run by using fewer bits to represent their weights. While unchanged answers on visual question answering (VQA) after compression are an expected behavior, they can still hide changes in the underlying scores (log probabilities). For example, a model may still answer yes after compression, even as the score gap between yes and no shrinks. We introduce BEFORE THE FLIP to measure these hidden changes. Our method compares the score change caused by compression with the change caused by replacing the image's internal representations, or image tokens, with one fixed average token. We then increase the precision of one weight group at a time to identify where extra bits help, and test whether choosing different groups for each question offers benefits beyond shuffled controls. Among 8,277 LLaVA questions where image token replacement measurably affects the scores, 4-bit compression shifts the yes or no score gap farther toward the replacement output than 8-bit compression. Qwen shows the same pattern, but with a smaller difference. Yet only 265 of 9,000 LLaVA answers change at 4 bits. In a separate study of 1,024 calibration questions, choosing weight groups separately for each question does not outperform both shuffled controls at any tested storage budget. These findings show that compression can alter the scores behind unchanged answers, but do not establish a reliable benefit from adjusting precision for each question.

1 Introduction

VLM quantization aims to reduce storage and runtime cost, but benchmark answers can conceal shifts in the scores that determine them. BEFORE THE FLIP measures these shifts and tests where extra precision helps and whether per-question precision selection is reliable.

1 Introduction

  • BEFORE THE FLIP compares compression-induced score changes with those caused by replacing image tokens with a fixed average token.A positive Prior Reversion Score means compression moves YES and NO scores toward the replacement output; it does not establish a hidden language prior.Source: §1 · p. 2
  • The method raises one of 57 weight groups from 4 to 8 bits at a time to identify where added precision reduces loss.The groups cover image encoder blocks, a connector, and decoder blocks.Source: §1 · p. 2
  • Per-question precision selection is tested against two shuffled controls, and its observed gain remains below at least one control at every tested storage budget.The paper also evaluates two VLMs, five image-replacement methods, captioning, four additional tasks, fixed plans, learned selection, and visual contrastive decoding.Source: §1 · p. 2
  • W4 raises mean PRS from 0.0094 under W8 to 0.0760 under W4, while only 265 of 9,000 LLaVA answers change.Qwen shows the same score-shift ordering, but less strongly; per-question selection does not reliably beat shuffled controls.Source: §1 · p. 2

2 Related Work

Prior work studies quantization methods for language and vision-language models, image-grounding and hallucination tests, and input-dependent decisions about model use. This paper connects these lines by measuring score shifts under controlled image replacement and testing query-dependent precision.

Quantizing language models

  • Language-model quantization methods reduce weight storage through approaches including quantization-error or activation-based choices, higher-precision outlier handling, and specialized weight representations [39, 40].These methods aim to reduce cost while preserving model performance.Source: §2 · p. 3

Quantizing vision language models

  • VLM quantization studies address differing image and text feature scales, identify sensitive model parts, or assign different precision levels across model parts [47].Related query-based decisions in QAQ and RouteLLM motivate testing whether precision should depend on the question [50] [51].Source: §2 · p. 3

Testing whether an answer follows the image

  • Grounding and hallucination evaluations include paired-image and controlled-example tests, while VCD and M3ID compare outputs from original and modified images [59].This work instead measures how low-bit weights move two answer scores under controlled image replacement.Source: §2 · p. 3

Choosing when to use a stronger system

  • Selective prediction, confidence calibration, and deferral study when systems should abstain, calibrate confidence, or hand off difficult cases [69].Here, per-question precision selection must outperform two shuffled controls, which preserve either group values or question difficulty.Source: §2 · p. 3

3 BEFORE THE FLIP: Our Proposed Three Stage Evaluation

BEFORE THE FLIP evaluates hidden score movement under quantization, the effects of increasing precision for individual weight groups, and whether question-specific group choices outperform shuffled controls.

3.1 Stage 1: Measure the Hidden Score Shift

  • Stage 1 compares the score shift from compression with the shift from replacing image tokens by a fixed average token, using two score-change vectors measured against the original 16-bit output.The replacement keeps the number and positions of tokens, question, and prompt unchanged; alternative replacement methods are also tested.Source: Figure 2 · p. 3, §3.1 · p. 4

3.2 Stage 2: Change One Weight Group

  • Stage 2 starts with 57 groups at 4 bits and raises one group at a time to 8 bits, measuring loss that combines added correct-answer cross-entropy error and positive PRS.A positive group improvement means raising that group to 8 bits lowers loss, but does not identify where visual grounding is stored.Source: Figure 2 · p. 3, §3.2 · p. 4

3.3 Stage 3: Test a Choice for Each Question

  • Stage 3 compares per-question hindsight group selection with two shuffles: one breaks question identity within each group, and the other breaks group identity within each question.The real hindsight gain must exceed both shuffled gains, and selected multi-group plans are run directly because group effects may interact.Source: Figure 2 · p. 3, §3.3 · p. 4

3.1 Stage 1: Measure the Hidden Score Shift

  • A positive PRS means compression moves the two answer scores toward the image-replacement output; questions with small replacement effects are filtered using a threshold.The method also reports PRS+, response ratio, and flip match rate.Source: §3.1 · p. 4

4 Experiments

Experiments test whether quantization shifts yes/no scores toward average-token outputs, where higher precision reduces loss, and whether precision can be selected per question; they also assess transfer across models, tasks, image replacements, and decoding methods.

4.1 Experimental Setup

  • The study evaluates LLaVA-1.5-7B and Qwen2.5-VL-7B-Instruct on POPE and additional caption, perception, and text tasks, with group selection analyzed on 1,024 calibration questions.The primary PRS analysis filters questions with negligible responses to image replacement, while precision-plan results include all 9,000 POPE questions.Source: §4.1 · p. 5

4.2.1 Quantization Shifts Scores Before Answers Change

  • Across 8,277 filtered questions, mean PRS rises from 0.0094 under W8 to 0.0760 under W4, while only 265 of 9,000 LLaVA answers change at W4.The measured PRS also depends on the image replacement; the five replacements produce different directions and sizes of shift.Source: Table 1 · p. 5, §4.2.1 · pp. 5–6

4.2.3 Per Question Selection Does Not Beat Both Shuffled Controls

  • At 5%, 10%, and 15% storage budgets, per-question gains fail to exceed both shuffled controls; group rankings across question halves correlate only r = .307.At the 10% budget, the observed gain of 1.3535 lies between shuffled gains of 1.6616 and 1.2309.Source: Figure 3 · p. 6, §4.2.3 · p. 6

4.2.2 Higher Precision Helps Earlier Decoder Blocks on Average

  • Across decoder blocks, depth correlates negatively with loss reduction (ρ = −0.633); first-half mean reduction is 0.00994 versus −0.00098 in the second half.Thirteen decoder blocks increase measured loss when raised to W8, and group interventions do not identify where visual grounding is stored.Source: §4.2.2 · p. 6, Figure 3 · p. 6

4.3.1 Fixed and Learned Precision Plans

  • The fixed 10% plan reduces PRS+ by 0.0650 relative to W4, while all three learned selectors choose K3_P2 for every held-out calibration question.The fixed plan’s PRS+ difference from the image-encoder rule has a 95% interval that includes zero.Source: Table 2 · p. 7, §4.3.1 · p. 7

4.3.2 Tasks, Models, and Decoding Settings

  • W4 changes five task averages by less than one point relative to 16-bit; on fixed-resolution Qwen2.5-VL, mean PRS rises from 0.00017 under W8 to 0.01397 under W4.The VCD setting with the lowest mean PRS does not produce the highest F1.Source: §4.3.2 · p. 7, Figure 4 · p. 8

5 Conclusion

BEFORE THE FLIP measures score changes that can remain hidden while VQA answers stay stable. Compression shifts scores toward the average-token output more at 4 bits than at 8 bits, but per-question precision selection does not outperform both shuffled controls.

5 Conclusion

  • W4 moves the YES–NO score margin farther toward the average-token output than W8 on LLaVA; Qwen shows the same ordering with a smaller effect.Only 265 of 9,000 LLaVA answers change at 4 bits.Source: §5 · p. 8
  • Raising individual weight groups to 8 bits benefits earlier decoder blocks on average, although parts within some blocks have opposite effects.Source: §5 · p. 8
  • Per-question precision gains fail to exceed both shuffled controls at any tested storage budget; selectors converge to one profile and invoke VCD for every question.The fixed 10% plan does not outperform the image encoder rule.Source: §5 · p. 8

6 Limitations

The experiments use low-precision weights with 16-bit operations and do not report latency, memory, or energy measurements. PRS is limited to binary answers and a replacement favoring NO; mean PRS lacks image-level confidence intervals.

6 Limitations

  • Experiments use low-precision weights with 16-bit operations, and report no latency, memory, or energy measurements.Source: §6 · p. 8
  • PRS is limited to binary answers and an image replacement that favors NO.Source: §6 · p. 8
  • Mean PRS is reported without image-level confidence intervals, and future work should examine additional VLMs.Source: §6 · p. 8

A Detailed Results and Definitions

The appendix defines the score measures and model components used in its tests, and distinguishes the question populations used for different analyses.

A Detailed Results and Definitions

  • PRS compares score movement from lower precision with movement from image replacement; positive values point toward the replacement output, zero means no such movement, and one reaches it along that direction.PRS+ retains only positive movement; faithfulness loss adds PRS+ to any increase in the correct answer's penalty, with lower loss better.Source: A Detailed Results and Definitions · p. 15
  • The vision encoder turns pixels into image tokens, the connector maps them into the language model, and the decoder produces the answer.CI means confidence interval.Source: A Detailed Results and Definitions · p. 15
  • W4 and W8 denote 4-bit and 8-bit weight storage, while the 16-bit model is the reference; a precision profile is one fixed plan for assigning weight bits.Source: A Detailed Results and Definitions · p. 15
  • Filtered PRS uses questions with measurable image effects, whereas all-question policy comparisons retain all 9,000 POPE questions; the group and selection studies use a separate pool of 1,024 calibration questions or its held-out parts.Filtered PRS must not be compared with all-question PRS+ or faithfulness loss.Source: A Detailed Results and Definitions · p. 15

A.1 How the Score Depends on the Image-Replacement Test?

Across POPE analyses, W4 PRS stays above W8, but its measured size depends on the stability filter and image-replacement test. Changed answers do not show significant reversion toward the replacement answer.

A.1 How the Score Depends on the Image-Replacement Test?

  • Filtered mean PRS is 0.07596 for W4 versus 0.00940 for W8; all-question means are 0.04582 versus 0.00081 (Table 3).The tenth-percentile stability cutoff retains 91.97% of questions; W4 PRS+ is 0.12701 filtered and 0.24101 on all questions.Source: Table 3 · p. 15, §A.1 · p. 15
  • Across fifth-, tenth-, and twentieth-percentile cutoffs, W4 PRS remains positive and above W8 for every POPE question type.Stricter filtering changes the retained share from 94–96% to 81–87%, and changes PRS values without changing the ordering.Source: Table 4 · p. 15, §A.1 · p. 15
  • Four post-connector token replacements yield positive W4 PRS on all three question types, while gray-image replacement yields 0.004, −0.042, and −0.039.The result applies to replacement after the connector, not every way of changing image information; wrong-image tokens yield PRS 0.043 for all three types.Source: Table 5 · p. 16, §A.1 · p. 16
  • Average-token replacement produces YES answers on only 0.6% of POPE questions, so positive PRS indicates movement toward that specified output, not an identified language prior.The replacement run has F1 1.4; zero tokens produce the opposite extreme and answer YES every time.Source: §A.1 · p. 16
  • W4 changes 265 of 9,000 answers, but 0.4264 of those changes match the 16-bit replacement answer, statistically unresolved against the shuffled rate of 0.4269 (p = 0.7225).W8 changes 22 answers, also unresolved against its shuffle control; the score ordering does not establish final-answer reversion or weaker image response.Source: §A.1 · p. 16

A.2 Which Model Parts Respond to Extra Precision

Extra precision affects model parts unevenly: vision blocks offer stronger recovery per stored byte, while decoder effects vary by depth and subpart. These rankings are not confirmed as stable causal orderings across prompts.

A.2 Which Model Parts Respond to Extra Precision

  • Vision blocks give more recovery per stored byte; decoder recovery falls with depth (ρ = −0.633, p = 0.0001), while vision depth shows no resolved trend (ρ = −0.133, p = 0.5354).Decoder first-half mean recovery is 0.00994 versus −0.00098 in the second half; vision half means are 0.01191 and 0.00946.Source: §A.2 · p. 16
  • Subpart gains can oppose each other and do not add exactly: block 1 gains through its FFN, while block 6 gains through attention.Block 1 has −0.0001 attention and 0.0275 FFN; block 6 has 0.0217 attention and −0.0039 FFN, and the mean absolute mismatch is 0.00203.Source: §A.2 · pp. 16–17, Table 7 · p. 17
  • In block 13, both subparts lower loss, yet their separate gains sum to 0.0052 less than the whole-block gain; the mean absolute mismatch is 0.00203.Source: §A.2 · p. 17
  • A whole-block change lowers loss in ten tested decoder blocks; in five, one subpart supplies at least 90% of the drop, reaching about 4.7 times more loss reduction per stored byte.The best decoder subpart lowers loss by about 0.0003 per million weights, versus about 0.0023 for a leading image-encoder block, so plans still choose image-encoder blocks first.Source: §A.2 · p. 17, Table 7 · p. 17
  • The calibration ranking is not confirmed as stable or causal: the two data halves correlate only 0.307, and confirmation would require filtered testing on new calibration images.The loss combines scaled PRS+ with any increase in correct-answer negative log probability relative to 16-bit, rather than using the filtered PRS in Table 3.Source: §A.2 · p. 17

A.3 Does the Best Choice Vary by Question?

Per-question hindsight choices appear advantageous, but they do not beat both shuffled controls at any tested storage budget. The measured question-specific variation is largely unresolved from noise, and tested selectors do not identify a usable plan under the W8-based limit.

A.3 Does the Best Choice Vary by Question?

  • At every tested storage budget, per-question hindsight gains fail to exceed both shuffled controls (Table 8).The controls preserve group-wise or question-wise score distributions while removing the tested question-to-group relationship.Source: Table 8 · p. 17, §A.3 · p. 17
  • Question-by-group variation plus measurement noise explains 93.5203% of recovery variance; the globally best group is best for only 0.879% of questions.Top-five group overlap averages 0.131, and Kendall rank agreement is τ = 0.004, where zero means no consistent ordering.Source: §A.3 · pp. 17–18
  • All three selectors assign K3_P2 to every held-out example, yielding 0% coverage at the W8-based limit; their profile-wise prediction errors span 0.429–0.642.Coverage is the share served without VCD; every upper prediction exceeds the limit.Source: Table 9 · p. 18, §A.3 · p. 18
  • Loosening the loss limit includes more questions even with hindsight, but does not show W4 features are informative or satisfy the original W8 standard.Selector errors of 0.43–0.64 are about as large as the W4-to-W8 loss span, 0.55 to 0.07.Source: §A.3 · p. 18, Figure 6 · p. 18

A.4 Fixed Precision Plans and General-Skill Evaluation

Fixed mixed-precision plans reduce measured calibration loss, while W4 already meets the reported general-skill criterion and answer and caption changes remain unresolved. The tested evidence does not establish a benefit from per-question precision selection or selective VCD.

A.4 Fixed Precision Plans and General-Skill Evaluation

  • W4 changes average general-skill accuracy by +0.05 percentage points, with no statistically resolved violation; the smallest valid protection set is empty.The required capstep01 procedure changes vision block 13 to W8, but the general-skill criterion does not require that group.Source: §A.4 · p. 19, Table 12 · p. 20
  • On held-out calibration examples, capstep01 lowers faithfulness loss by 0.0318 from W4, and K3_P3 lowers it by another 0.1627.Replacing K3_P3 with the fixed 10% average-token plan changes loss by only −0.0025; the per-question hindsight value of 0.1847 is described as inflated by noise.Source: Table 10 · p. 19, §A.4 · p. 20

A.5 Do Any Answers or Captions Improve?

  • Across 12 precision plans, POPE accuracy differences are unresolved, while the fixed 10% average-token plan lowers all-question PRS+ from W4 by 0.0650.Its adversarial accuracy change is −0.10 points (95% CI [−0.70, +0.50]); Table 13 reports PRS+ and loss over all 9,000 questions.Source: Table 13 · p. 20, §A.5 · p. 20
  • For 500 COCO captions, CHAIRs is 24.0% for the 16-bit model and 24.6% for W4, an unresolved paired difference of −0.60 points (95% CI [−4.00, 2.80]).K3_P3 has the lowest rate at 23.2%, also unresolved versus W4; Table 14 reports paired image-level resamples.Source: §A.5 · p. 20, Table 14 · p. 21
  • Qwen repeats the W4-over-W8 PRS ordering with a smaller shift: 0.01397 versus 0.00017, but mixed-precision plans are not tested on Qwen.Qwen uses 256 visual tokens versus LLaVA’s 576, so the experiment cannot separate model-design effects from token-count effects.Source: Table 15 · p. 21, §A.5 · p. 21

A.6 Contrastive Decoding, Examples, and Storage Estimates

  • With 999 corruption steps, raising VCD α from 0 to 1 moves all-question PRS from 0.0458 to −0.6884, while accuracy falls from 85.92% to 83.19%.F1 changes by −0.32 points, and PRS+ rises from 0.241 to 1.243; PRS and PRS+ favor different settings.Source: Table 16 · p. 21, §A.6 · p. 21
  • Selective VCD is used on 100% of questions because no calibrated upper prediction falls below the loss limit, making its outputs identical to always-on VCD.The experiment therefore tests an always-on two-image correction, not selective computation.Source: Table 17 · p. 22, §A.6 · p. 22
  • Only 66 of 9,000 POPE questions (0.73%) follow the complete answer-flip pattern; the fixed K3_P2 profile restores 45 of those 66 cases (68.2%).Selectors also assign K3_P2 to every question, so the restoration is not attributable to routing.Source: §A.6 · p. 22

Related papers

Loading 2609.06922v1…