MAGIC: Margin-gated Contrastive Debiasing with Evidence Restoration for VLM Test-time Robustness Scaling

1 Introduction

• LVLMs can favor common language-prior answers over what an image shows, motivating inference-time debiasing that requires no retraining.The passage notes that this correction applies to any pretrained model.Source: §1 · p. 1
• Contrastive-decoding corrections use corrupted branches, but can discard genuine image evidence and spend forward passes on every input.The passage also notes diminishing returns from adding branches because averaging only shrinks part of the prior-estimation error.Source: §1 · p. 1
• MAGIC gates correction by decision margin, leaving confident predictions untouched and directing correction to contested ones.It is presented as a training-free corrector with no trained parameters.Source: §1 · pp. 1–2
• MAGIC restores evidence from the perturbed view where the model is most confident, retaining it only when the decision becomes more definite.Its perturbation branches are pruned per model using a calibration set rather than fixed in advance.Source: §1 · pp. 1–2
• MAGIC consistently outperforms strong contrastive-decoding baselines across four LVLMs from two generations, with inference cost that adapts to the input.The introduction also reports evaluation across six datasets.Source: §1 · p. 2

Large Vision-Language Models

• LVLMs combine visual encoders and pretrained language models, whose language priors can dominate answers even when images show otherwise.The passage says differences in encoders, connectors, and training data may mean models need different corrections.Source: §2 · p. 2

Test-Time Robustness Scaling

• Contrastive-decoding methods estimate bias by comparing an original run with a weakened or image-corrupted run, using variants such as VCD, M3ID, and SCI.SCI averages multiple perturbation branches; the passage also describes counterfactual-inference methods as reaching the same subtraction through causal effects.Source: §2 · p. 2
• Prior adaptive methods vary when or which perturbation to use, but their branch sets remain fixed and are charged to every sample; MAGIC also restores removed visual evidence and prunes branches per model.The passage distinguishes adjacent approaches that act within the forward pass, at training time, or through prompt design.Source: §2 · p. 2

3.1 Framework Overview

• MAGIC treats correction as a resource: confident predictions keep one forward pass, while contested predictions receive debiasing and may receive a visual residual.Zero gates recover the base model; an always-open first gate with one blank branch reduces MAGIC to the earlier corrector.Source: §3.1 · p. 3

3.2 Debiased Contrastive Decoding with Margin Gating

• On natural samples, debiasing becomes harmful at small margins, while on biased subsets it helps across a wider margin range, supporting margin-gated correction.The same utility pattern appears across all four backbones.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Selective Evidence Restoration for Discriminative Decision-Making

• For contested debiased states, MAGIC selects the perturbed view with the sharpest prediction distribution and restores its visual residual only when it widens the margin enough.A residual that fails the threshold is discarded, leaving the debiased state unchanged.Source: §3.3 · p. 4

3.4 Perturbation Branch Pruning with Calibration

• Per-model calibration retains perturbation families that improve calibration accuracy, because averaging leaves a bias floor and each added branch’s variance reduction vanishes.The thresholds are fitted on the same validation split used as the calibration set.Source: §3.4 · p. 5

4.1 Experimental Settings

• The evaluation covers DRBench’s B, S, and BS subsets plus six real-world datasets, using four backbones and reporting accuracy and average forward passes per sample.The visual counterfactuals include a blank image, grayscale, blur, and mean-color masking; the answer-format branch changes the wording while retaining the image.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 Performance Comparison

• On the biased subset of Qwen2-VL, MAGIC traces an accuracy–compute frontier as t1 varies, whereas each baseline has a fixed budget.The star marks the deployed setting.Source: Figure 4 · p. 7
• 18.61% and 17.47% higher BS Overall accuracy than the base model on Qwen2-VL and LLaVA-NeXT, respectively; MAGIC also exceeds M3ID by 7.45% and 7.17%.On DRBench, MAGIC improves over cost-matched SCI5 by 3.63% and 2.03%, and exceeds SCI7 on Overall using roughly 70% of its passes.Source: §4.2 · pp. 6–7, Table 2 · p. 6
• On six real-world LVLM datasets, MAGIC improves over the base on both backbones and exceeds the strongest CD baseline’s Overall at fewer passes.The reported margins are small because vulnerable samples make up only a fraction of these datasets; TIE and VCD lose accuracy on the Others question type.Source: Table 3 · p. 6, §4.2 · p. 7
• 19.28% and 15.58% higher BS Overall accuracy than the base on Qwen3-VL and LLaVA-OneVision-1.5, respectively; MAGIC also beats the best SCI setting by 5.40% and 0.92%.On these newer backbones, calibration retains one debiasing contrast on each, and drops restoration on Qwen3-VL; the correction costs 2.9 and 4.0 passes.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Further Analysis

• Removing either gated step hurts performance, while using only one debiasing contrast instead of both costs 0.5%–1.7% in BS Overall.On Qwen2-VL, raising t1 above 3.25 gains at most 0.4% in BS Overall, while t2 moves BS Overall by 1.98% as mean passes rise from 4.0 to 5.5.Source: §4.3 · pp. 7–8, Table 5 · p. 8, Table 6 · p. 8
• The two case studies show input-dependent compute: MAGIC spends four passes when debiasing widens the margin, and seven when restoration is also triggered.In the latter case, the visual residual adds 0.65 to the correct option; the figure marks each stage’s gate and the committed answer.Source: §4.3 · p. 8, Figure 5 · p. 8

• MAGIC fires only where the decision margin is small, restores visual evidence removed by perturbation, and sizes its branch set per model.The paper describes MAGIC as a training-free corrector for test-time robustness scaling.Source: §5 · p. 8

• Because MAGIC scores a fixed candidate set rather than intervening during token-by-token decoding, free-form captioning benchmarks fall outside these results.Its three thresholds are fitted per target distribution on a validation split, so a new distribution requires rerunning that selection.Source: Limitations · p. 9

• On Qwen2-VL and LLaVA-NeXT, calibration sets t1 to 3.25 and 5.03; t2 and τV vary by backbone and answer format, while full-distribution fits lower t1 to 0.5 and 2.0.Fitting replays cached branch scores and requires no additional model passes.Source: A Implementation Details · p. 11

B.1 Inertness of a Bounded Correction

• When the top-two margin m exceeds 2B, a correction bounded by B cannot change the argmax; the same bound applies to restoration.The bound is worst-case, so the debiasing gate is instead fitted using the measured utility curve.Source: §B.1 · p. 11

B.2 The Bias Floor and the Branch Budget

• Perturbation-induced bias leaves a nonzero error floor, while each added branch’s maximum marginal error reduction vanishes, yielding a finite branch budget.The finite optimum depends on encoder noise and any growth in bias increments.Source: §B.2 · p. 11
• The comparison reports 0.57% of error on LLaVA-NeXT; it also says variance above the floor is encoder-specific, while the Qwen2-VL error figure is not given in this passage.Source: §B.2 · p. 12

C.1 Additional Case Studies

• Four traced exits show MAGIC skipping confident cases, accepting debiasing alone, using restoration to flip an answer, or rejecting a residual that would undo a repair.The rejected residual would move the answer to sadness with a narrower margin.Source: §C.1 · p. 12

C.2 Full Sensitivity Analysis

• Above fitted t1=5.03, LLaVA-NeXT Overall gains at most 0.25% through t1=8; t2 moves BS Overall by 0.52% as mean passes rise from 4.0 to 5.5.The reported pattern matches Qwen2-VL’s threshold sweep.Source: §C.2 · p. 12

C.3 Cost Analysis

• On MMStar, MAGIC uses 1.98 passes on Qwen2-VL and 3.47 on LLaVA-NeXT versus SCI5’s five; latency is 4.57 s vs 4.58 s and 0.69 s vs 0.65 s, respectively.The gate fires on about a quarter of MMStar samples; on biased subsets, almost every sample is contested.Source: §C.3 · p. 12

C.4 Perturbation Invariance

• Random and keyword-matched real-photo replacements behave like blank images at decision, while keyed-full is the answer-producing positive control.The comparison measures decision invariance over replacements that do not answer the question, not distributional invariance.Source: §C.4 · p. 12, Figure 7 · p. 13

C.5 Attribution Analysis

• VACoDe and CHASD beat MAGIC on no cell; keeping the gate but dropping the second contrast and residual costs 3.0% BS Overall on Qwen2-VL and 1.1% on LLaVA-NeXT.The table compares answer-level adaptations on MCQ cells of the test split.Source: §C.5 · p. 13, Table 9 · p. 14
• At matched firing rates, replacing the margin with maximum softmax probability or predictive entropy shifts BS Overall by at most 0.08%; adding the gate to SCI leaves calibration predictions unchanged.The authors conclude that the correction, not the gate measure, separates MAGIC from these comparisons.Source: §C.5 · p. 13

D Prompt Templates

• Table 10 pairs answer-format rephrasings with representative benchmark instructions; variants of the same instruction map to the same rephrasing.The pairs are matched on the instruction issued by the benchmark.Source: D Prompt Templates · p. 14, Table 10 · p. 15