PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models

Minchan Kang, Kyeonghye Park, Seungyeon Sa, Seoyoung Cho, Daeshik Kim, Yucheol Cho

arXiv:2609.34765v1cs.CVcs.LG

TL;DR

LVLM PTQ methods use limited calibration data but still optimize reconstruction against the full-precision model, despite heterogeneous quantization effects across components. Balanced Fitting adapts fitting strength to those effects, and experiments report consistent gains over prior methods across tasks and quantization settings.

  • Problem

    LVLM PTQ methods incorporate sensitivity signals but still minimize reconstruction error against the full-precision model on small calibration sets that may not represent downstream data.

    Source: §1 · p. 1

  • Method

    Balanced Fitting measures layer- and component-wise quantization effects and allocates fitting strength to fit sensitive components finely while coarsening fitting where quantization may regularize.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    Balanced Fitting consistently outperforms prior PTQ methods across five-task averages; on MMMU, accuracy-drop reduction averaged 2.43× versus the strongest competing baseline.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Lower reconstruction loss does not necessarily correspond to better downstream performance, making it an unreliable proxy for evaluating quantized LVLMs.

    Source: §4.3 · pp. 8–9, §5 · p. 9

Abstract

from arXiv · show

Post-training quantization (PTQ) enables efficient deployment of large vision-language models (LVLMs), but is typically calibrated on a small set while expected to generalize across diverse downstream tasks. Although recent PTQ methods for LVLMs incorporate sensitivity signals, they still minimize reconstruction loss with respect to the full-precision model, potentially over-preserving FP behavior and calibration-specific bias. Rather than treating quantization solely as an error to be minimized, we observe that it can also provide beneficial regularization for certain layers and modalities. Motivated by this observation, we propose Balanced Fitting, a quantization effect-based framework that balances precision and regularization beyond reconstruction-based optimization. By measuring layer- and component-wise quantization effects for weights, vision activations, and text activations, Balanced Fitting combines fine-grained fitting for sensitive components with coarser fitting to exploit potential regularization benefits. Experiments on multiple LVLMs show that our method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance. The source code is publicly available at https://github.com/kmc3661/BFQ

1 INTRODUCTION

LVLM PTQ methods use sensitivity signals but still optimize reconstruction against the full-precision model on small calibration sets. Balanced Fitting adapts fitting strength to component-wise quantization effects, and experiments report consistent gains over prior methods.

1 INTRODUCTION

  • Prior LVLM PTQ methods use sensitivity signals but still minimize reconstruction error against the full-precision model on a small calibration set.The limited calibration set may not reflect downstream data, while reconstruction fitting can preserve overfitted full-precision behavior and calibration-set bias.Source: §1 · pp. 1–2
  • Quantization can act as an implicit regularizer in some layers or components, while other components are sensitive to it.The paper relates this heterogeneity to the possibility that full-precision models contain overfitted or suboptimal components.Source: §1 · pp. 1–2
  • Balanced Fitting measures quantization effects for weights, vision activations, and text activations, then fits sensitive components more finely and coarsens fitting where quantization may regularize.The framework adapts fitting capacity using component-aware scores.Source: §1 · p. 2
  • On MMMU, Balanced Fitting reduces accuracy drop by 2.43× on average and up to 5.2× versus the strongest prior PTQ baseline across models and bit-widths.Figure 1 reports the lowest accuracy drop among state-of-the-art LVLM PTQ methods across models and W3A16 and W4A8 settings.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

The related work situates LVLMs as multimodal models with deployment costs and reviews PTQ approaches that address quantization sensitivity. It contrasts reconstruction-based calibration with Balanced Fitting’s component-aware fitting strength.

2.1 LARGE VISION LANGUAGE MODELS

  • LVLMs combine vision-encoder features with text tokens for multimodal reasoning, while this work focuses on efficient deployment through PTQ.Representative models include LLaVA-OneVision, InternVL, and Qwen-VL.Source: §2.1 · p. 2

2.2 POST-TRAINING QUANTIZATION

  • Prior LVLM PTQ methods use cross-layer dependencies, modality-wise gradients, or token importance, but retain reconstruction-based optimization on limited calibration data.Balanced Fitting instead adapts fitting strength to component-wise quantization effects, preserving sensitive components while allowing beneficial deviations from the full-precision model.Source: §2.2 · p. 3

3 METHOD

Balanced Fitting uses layer- and component-wise quantization effects to vary the precision of CWE search, fitting sensitive components more finely and using coarser search where quantization may regularize. The method retains the reconstruction objective but allocates search capacity across layers according to their quantization effects.

3.1 PRELIMINARIES

  • PTQ maps weights and activations to low-bit values, and existing LVLM methods use channel-wise equalization to reduce activation outliers by inversely rescaling weights and activations.Balanced Fitting retains the quantization operator and reconstruction objective while changing the number of CWE search candidates per layer.Source: §3.1 · p. 3

3.2 LAYER- AND COMPONENT-WISE QUANTIZATION EFFECTS

  • Quantization effects are measured by selectively quantizing one component at a time in each layer and tracking the impact on calibration loss.The components are weights, vision activations, and text activations; measured effects vary across layers and components.Source: §3.2 · p. 4
  • Because component responses can disagree within a layer, the method incorporates a layer-wise conflict score into importance scores to avoid obscuring vulnerable components.Source: §3.2 · p. 5

3.3 LAYER-WISE BUDGET ALLOCATION

  • Layer-wise budgets allocate CWE grid points from quantization-effect importance, with a minimum per-layer budget and remaining capacity concentrated on higher-importance layers.The median score marks the lower-budget group; the concentration factor controls how strongly remaining budget is focused on higher-scoring layers.Source: §3.3 · p. 5
  • For W4A8, weight and activation branches contribute to the final search budget, whereas W3A16 uses only the weight branch.Source: §3.3 · p. 6

3.4 CWE WITH BUDGET-CONTROLLED SEARCH

  • CWE search budgets control fitting granularity: larger budgets enable finer search for sensitive layers, while smaller budgets enforce coarser search that may reduce overfitting.The method retains the standard CWE objective and changes the number of scale candidates searched per layer.Source: §3.4 · p. 6

4 EXPERIMENTS

Balanced Fitting outperforms prior PTQ methods across the evaluated models and settings. Ablations show that adaptive fitting, calibration stability, and domain-shift robustness are relevant to its performance.

4.1 EXPERIMENTAL SETUP

  • Experiments use 64 COCO Caption image–caption pairs for calibration and evaluate three LVLMs on five benchmarks, with an additional InternVL2-26B evaluation.The benchmarks cover visual reasoning, real-world perception, and visual understanding; InternVL2-26B is additionally evaluated on NoCaps-lite image captioning.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Balanced Fitting outperforms existing PTQ methods in five-task average across all models and both precisions, and reduces MMMU accuracy drop by 2.43× on average versus the strongest competing baseline.Table 2 also reports that it leads in average performance for InternVL2-26B under W4A8.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • Qualitative examples show Balanced Fitting preserving correct full-precision answers while correcting erroneous predictions reproduced by MBQ and QIG.The examples are from MMMU and VizWiz.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 ABLATION STUDY AND ANALYSIS

  • Adaptive allocation consistently outperforms uniform budgets, and uniformly assigning high budgets can underperform lower-budget configurations.Lower reconstruction loss also does not necessarily correspond to better downstream performance.Source: §4.3 · p. 8, Table 4 · p. 8
  • Full Balanced Fitting achieves the best average among refinement, coarsening, and their combination, while Low-only also beats the uniform baseline.These variant comparisons indicate a regularization effect from constrained fitting.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Quantization effects show strong held-out sign agreement for high-magnitude effects, and layer-wise profiles remain consistent across calibration sample sizes.Under a COCO Caption-to-Flickr30k calibration shift, Ours retains the highest W4A8 five-task average across all three models.Source: §4.3 · p. 9, Table 5 · p. 9, Table 6 · p. 9

5 CONCLUSION

The conclusion argues that reconstruction-loss minimization on small calibration sets is insufficient for LVLM generalization. Balanced Fitting instead allocates layer-wise budgets to balance precision and regularization, and experiments show consistent improvements over prior PTQ methods.

5 CONCLUSION

  • Minimizing reconstruction loss on small calibration sets is insufficient for LVLM generalization.The conclusion identifies reconstruction loss as a limited proxy for downstream performance.Source: §5 · p. 9
  • Balanced Fitting uses quantization effects to allocate layer-wise budgets, balancing precision and regularization across components.The framework is presented as an alternative to reconstruction-based optimization.Source: §5 · p. 9
  • Experiments demonstrate consistent improvements over prior PTQ methods and reveal that reconstruction loss is limited as a proxy for downstream performance.Source: §5 · p. 9

A ADDITIONAL ANALYSIS

Quantization effects differ across layers, components, models, and bit-widths, motivating effect-guided rather than fixed allocation. The reported analyses examine combined refinement and coarsening, and the stability of effect estimates across sample sizes.

Component-wise Quantization Effects Across Models and Settings

  • Quantization effects vary across layers, components, models, and bit-widths, so sensitive and beneficial components occur in different places across settings.Negative effects indicate increased calibration loss; positive effects indicate reduced loss and potential regularization benefits.Source: A ADDITIONAL ANALYSIS · p. 13

Effect of Balanced Budget Allocation on Downstream Performance

  • Under W4A8, the comparison supports combining refinement and coarsening rather than relying on either alone.Under W3A16, the minimum grid size of 20 makes High-only equivalent to Full and Low-only equivalent to the uniform baseline, so the setting tests refinement above baseline rather than a separate coarsening effect.Source: A ADDITIONAL ANALYSIS · p. 13

Calibration-Set Size Stability

  • Layer-wise quantization-effect profiles remain highly correlated across 16, 32, and 64 calibration samples across three LVLMs under W3A16 and W4A8.This complements held-out sign-agreement tests across calibration subsets.Source: A ADDITIONAL ANALYSIS · p. 13

B QUANTIZATION EFFECT-GUIDED ALLOCATION RULE

The supplementary allocation rule maps quantization-effect statistics to settings automatically, with performance close to empirically selected configurations. It uses distinct statistics for W3A16 and W4A8, and links allocation choices to loss concentration and activation-side loss.

B QUANTIZATION EFFECT-GUIDED ALLOCATION RULE

  • The statistics-driven rule automatically selects allocation settings from quantization-effect distributions and achieves performance close to empirically selected configurations.It uses the same formulas and normalization constants across models within each quantization mode.Source: B QUANTIZATION EFFECT-GUIDED… · p. 13
  • For W3A16, only the weight component is used; for both quantization modes, concentration is measured over the top 10% of layers with positive loss mass.Source: B QUANTIZATION EFFECT-GUIDED… · p. 15
  • For W4A8, the rule aggregates all components and accounts for activation-side loss share; broader, stronger activation-side loss increases the target budget.The rule also links the minimum budget to loss concentration and activation-loss spread.Source: B QUANTIZATION EFFECT-GUIDED… · pp. 15–16
  • The rule normalizes statistics using mode-specific bounds shared across models, clips values outside those bounds, and rounds settings to the nearest multiple of 0.05.The bounds are specified separately for W3A16 and W4A8.Source: B QUANTIZATION EFFECT-GUIDED… · p. 15
  • The rule’s selected settings are close to the best observed configurations, though they do not always maximize MMMU.Figure 7’s W3A16 comparison excludes High-only and Low-only because they coincide with Full and the baseline, respectively.Source: Figure 7 · p. 14, B QUANTIZATION EFFECT-GUIDED… · p. 16

C.1 EVALUATION SETTINGS

The appendix tests Balanced Fitting against alternative quantization baselines, allocation strategies, calibration budgets, and runtime settings. Across these comparisons, it reports stronger five-task averages in the tested settings, with calibration-time advantages dependent on hardware allocation.

C.1 EVALUATION SETTINGS

  • For InternVL2-26B, the common rule yields gmin = 14, gavg = 21, and γ = 1.8; Flickr30k uses 64 calibration pairs and fixed 500-example splits for AI2D-lite and NoCaps-lite.The evaluation also sweeps base grid, target mean, and γ, reporting five-task averages and MMMU performance.Source: §C.1 · p. 16, Figure 9 · p. 17

C.2 LLM-BACKBONE QUANTIZATION BASELINES

  • Ours achieves a higher five-task average than LLM-backbone AWQ and GPTQ on all three models.The comparison applies AWQ and GPTQ to the LLM backbone under W3A16 while leaving the visual encoder in full precision.Source: Table 7 · p. 18, §C.2 · p. 18

C.3 SENSITIVITY-MATCHED MIXED PRECISION

  • Ours outperforms sensitivity-matched mixed precision and uniform CWE-20 across all three models and both target budgets.The comparison concerns the tested mixed-precision control; Balanced Fitting retains uniform inference precision.Source: Table 8 · p. 18, §C.3 · p. 19

C.4 QIG WITH A LARGER CALIBRATION SET

  • With half as many calibration pairs as QIG, Ours has the higher five-task average in five of six model–precision settings; Qwen2-VL W3A16 is the exception.This supports calibration-data efficiency, not wins on every individual task.Source: Table 9 · p. 19, §C.4 · p. 19

C.5 BLOCK-SENSITIVITY ALLOCATION CONTROLS

  • The independent fixed-grid block control uses grids 16, 18, 20, 22, and 24, while the grid-matched control preserves Ours' grid-size multiset and total search budget.The independent control has a comparable, but not identical, budget.Source: §C.5 · p. 19
  • Ours achieves the highest five-task average against both block-sensitivity controls on all three models.Relative to Fixed-grid Block, MMMU improves on every model by up to 2.55 points, while VizWiz improves on two models by up to 3.43 points.Source: Table 10 · p. 20, §C.5 · p. 20

C.6 COMPUTATIONAL COST

  • Ours has lower total elapsed time than QIG on all three models and than MBQ on LLaVA-OV, but MBQ is faster on the other two.The wall-clock comparison uses different GPU allocations; calibration is performed once and adds no inference overhead.Source: Table 11 · p. 20, §C.6 · p. 20

D.1 STABILITY METRIC DEFINITIONS

The stability analysis checks whether calibration-measured quantization effects retain their direction on held-out data. A controlled search-granularity test finds that selectively coarser fitting can outperform finer search, without showing a monotonic benefit from reducing candidates.

D.1 STABILITY METRIC DEFINITIONS

  • The stability test ranks components using calibration mean relative absolute QE, then compares calibration sign agreement with an independent held-out subset.Beneficial held-out agreement is restricted to calibration-beneficial effects.Source: §D.1 · p. 21
  • Agreement is strongest for high-magnitude quantization effects, including beneficial effects, supporting both refinement and coarsening decisions.This analysis tests allocation-signal stability, not variation in final downstream performance across calibration seeds.Source: §D.1 · p. 21

D.2 CONTROLLED SEARCH GRANULARITY

  • Coarse search achieves the highest five-task average on all three models despite highly similar Fine–Coarse scale vectors.The intermediate setting is not consistently better than Fine, supporting selective rather than monotonic restriction of fitting capacity.Source: Table 12 · p. 21, §D.2 · p. 21

D.3 FULL-SET FP-ANSWER TRANSITIONS

Full-set prediction transitions indicate that Balanced Fitting's aggregate gains chiefly reflect retaining correct full-precision behavior and avoiding new errors, rather than uniformly correcting more mistakes. Under joint quantization, isolated quantization effects remain informative but do not fully capture component interactions.

D.3 FULL-SET FP-ANSWER TRANSITIONS

  • The transition analysis evaluates 12,824 paired examples per model and precision across MMMU, VizWiz, ScienceQA, ChartQA, and AI2D.Correctness follows each benchmark's evaluator, and VizWiz positive VQA credit counts as correct.Source: §D.3 · p. 21
  • Ours consistently reduces new errors and improves retention and the fixes/new-errors ratio, while its correction rate is not uniformly highest.The full-set analysis evaluates paired FP–PTQ predictions across five benchmarks, with transition rates conditioned on FP correctness.Source: §D.3 · pp. 21–22, Table 13 · p. 22

D.4 QE UNDER JOINT QUANTIZATION

  • Restoration-based QE is the calibration-loss difference between restoring one component to FP and the jointly quantized model; positive values indicate lower loss with that component quantized.The comparison uses COCO-64 with allocation and CWE search disabled.Source: §D.4 · p. 22
  • Isolated and restoration-based QE show stronger sign agreement for high-magnitude effects, including beneficial effects, but agreement is imperfect.This supports using isolated measurements to identify influential components without fully capturing their interactions under joint quantization.Source: Table 14 · p. 22, §D.4 · p. 22

E ADDITIONAL QUALITATIVE RESULTS

Additional qualitative comparisons across three LVLMs and five benchmarks illustrate how Balanced Fitting handles incorrect full-precision predictions and ambiguous inputs. The reported pattern emphasizes preserving valid full-precision behavior while reducing newly introduced errors.

E ADDITIONAL QUALITATIVE RESULTS

  • Across the additional benchmark examples, Balanced Fitting more reliably preserves correct FP predictions and introduces fewer new errors, while also correcting some FP mistakes.The examples cover MMMU, VizWiz, ScienceQA, ChartQA, and AI2D across three LVLMs.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 23, Figure 11 · p. 24
  • On MMMU, selective coarse fitting can avoid reproducing an incorrect FP answer and recover the correct choice.On VizWiz, the examples include correctly identifying unanswerable cases where baselines often produce incorrect specific answers.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 23
  • The results suggest a balance between precise fitting and regularization-like capacity control, preserving valid FP behavior under aggressive quantization.The passage attributes the primary benefit to preserving valid behavior while reducing calibration-specific overfitting.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 23

Related papers

Loading 2609.34765v1…