Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Level

1 Introduction

• The controlled framework varies resolution, aspect ratio, detail, and task-relevant cue position while preserving semantics, helping isolate visual-encoding sensitivity from reasoning ability.The transformations are applied to existing VQA benchmarks to distinguish sources of performance degradation and identify architecture-specific susceptibilities.Source: §1 · pp. 1–2
• AUSC measures performance degradation as resolution increases, while PVS measures prediction instability across resolution variants of the same image.The metrics are designed to quantify resolution robustness independently of baseline accuracy.Source: §1 · p. 2
• Across five architecturally distinct model families, the paper identifies information loss from downsampling, tokenization artifacts, and attention dilution as three prominent failure modes.Fine detail may be discarded beyond vision-token budgets; patch misalignment can destabilize predictions; and larger token counts can dilute focus on relevant cues.Source: §1 · p. 2
• The onset and severity of the three failure modes vary by architecture, reflecting differences in dynamic resolution strategies, token-budget management, and attention mechanisms.The paper argues that standard single-resolution benchmarks are blind to these differences.Source: §1 · p. 2

2.1 Scaling Vision Encoders to High Resolution

• Fixed-resolution resampling discards information, while dynamic tiling adds computation and retains resolution and aspect-ratio constraints; native-resolution models instead incur variable token counts.Content-adaptive tokenization and selective encoding reduce computation but may introduce sensitivity to image layout and content position.Source: §2.1 · pp. 2–3

2.2 Evaluating High-Resolution Visual Processing

• High-resolution benchmarks examine fine-grained processing and spatial grounding, but generally evaluate images at a single resolution, confounding resolution effects with semantic complexity and task difficulty.The cited benchmarks cover dense scenes, small-region evidence extraction, counting, OCR, and spatial retrieval.Source: §2.2 · p. 3

2.3 Evaluating Robustness in Vision–Language Models

• Prior robustness studies identify architectural failure patterns but do not systematically isolate resolution, detail, and spatial sensitivity or attribute their effects to individual components.The reviewed findings connect sensitivity to geometric perturbations, encoder error accumulation, multimodal alignment, and uneven token compression.Source: §2.3 · p. 3

3.1 Problem Formulation

• The formulation treats a VQA benchmark as image–question/answer pairs and evaluates model predictions against ground truth under transformations that preserve task-relevant visual content.Standard fixed-resolution accuracy implicitly assumes performance is invariant to image scale and tokenization structure.Source: §3.1 · p. 3

3.2 Failure Modes

• The paper identifies three degradation sources across model components: preprocessing downsizing, visual-encoder tokenization artifacts, and LLM attention dilution.Evaluating across controlled transformations provides a structured view beyond aggregate accuracy and helps attribute likely failure modes to components.Source: §3.2 · p. 3, §3.1 · p. 3

3.2.1 Downsizing-Induced Information Loss

• Downsizing can discard fine-grained visual detail before encoding, including when a model’s vision-token budget is exceeded and the input is automatically downsampled or cropped.This constrains spatial detail available to downstream components even in models that support high-resolution inputs.Source: §3.2.1 · pp. 3–4

3.3 Tokenization Artifacts

• Patch-grid shifts can change which visual features are split across patch boundaries, while downsampling can raise semantic density per patch and alter the detail individual tokens must encode.Representations of the same content at different background locations become more similar in later vision-encoder layers, while positional differences remain encoded and location-dependent differences emerge in later decoder layers.Source: §3.3 · p. 4

3.4 Attention Dilution

• Padding native-resolution images increases token counts quadratically, spreading attention across redundant tokens and reducing the relative weight assigned to semantically meaningful regions.The figure illustrates attention diffusion across blank regions despite identical task-relevant visual content.Source: §3.4 · p. 4, Figure 2 · p. 4

4 Evaluation Framework

• Two transformation protocols separate full-pipeline responses to increased pixel density from tokenization effects while holding inter-pixel content fixed in the latter.The authors describe the difference in model response between the scenarios as causally informative.Source: §4 · p. 4

4.1 Pixel-Level Resolution Scaling

• Bicubic upsampling preserves global semantics and relative object scale, distributing the same content across more pixels while task-relevant detail stays constant.This probes changes in information density per token.Source: §4.1 · p. 5

4.2 Token-Level Resolution Scaling via Canvas Padding

• Canvas padding leaves image pixels unchanged but increases preprocessing demands, isolating tokenization effects including patch misalignment, longer token sequences, and attention dilution.Dynamic tiling models may also produce a different tile arrangement.Source: §4.2 · p. 5
• Top-left and center placement test patch alignment and spatial attention bias; top-left preserves original patch alignment for models that do not resize.Center placement tests spatial attention bias and tile-boundary splitting.Source: §4.2 · p. 5

4.3.1 Area Under the Scaling Curve (AUSC)

• AUSC uses accuracy across transformations on examples correct at original resolution, isolating scaling behavior from baseline performance.It is computed by the trapezoidal rule and ranges from 0 to 1, with 1 indicating perfect robustness and 0 complete brittleness.Source: §4.3.1 · pp. 5–6

4.3.2 Prediction Variance Score (PVS)

• PVS measures the fraction of initially incorrect predictions corrected by a transformation, capturing resolution-dependent instability.Values near 0 indicate stable errors that likely reflect reasoning limitations; higher values indicate resolution-dependent instability.Source: §4.3.2 · p. 6

4.4 Attribution Analysis

• Binary correctness sequences across scaling levels yield diagnostic answer-trajectory patterns whose cross-scenario fingerprints support attribution.The patterns are summarized in Table 1.Source: §4.4 · p. 6, Table 1 · p. 6

5.1 Experimental Set-up

• Experiments covered six named model series, five VQA benchmarks, 200 sampled instances per benchmark (1,000 total), scales 1.5×–6×, and hybrid answer parsing and LLM judging.The sampled benchmarks were MMVet, CharXiv, MME, MMStar, and MathVerse.Source: §5.1 · pp. 6–7

5.2.1 Accuracy Analysis

• Under 2D padding, degradation is modest (≤10%); top-left placement consistently underperforms centering, indicating a learned bias toward centered content.Native-resolution models are barely affected by 2D padding.Source: §5.2.1 · p. 7
• 1D padding causes 5–18% degradation despite fewer tokens than 2D padding; interpolation may improve accuracy at 1.5× before declining or degrade immediately.The authors identify positional encoding fragility under extreme aspect ratios as an independent failure mode.Source: §5.2.1 · p. 7

5.2.2 AUSC and PVS

• 1D padding produces larger AUSC degradation than 2D padding despite fewer tokens, with gaps reaching 20 points for Pixtral (0.521 vs. 0.707) and Llama (0.569 vs. 0.719).The authors interpret this contrast as evidence that positional encoding fragility is more damaging than attention dilution over uninformative padding regions.Source: Table 2 · p. 8, §5.2.2 · p. 8
• Pixtral combines the highest interpolation AUSC (0.890) with the lowest 1D padding AUSC (0.507), showing that scale and positional robustness are distinct axes.Qwen3-VL-8B shows the inverse pattern, with strong padding tolerance but interpolation stability of 0.683.Source: §5.2.2 · p. 8
• PVS values of 0.30–0.49 across conditions indicate substantial baseline-error recovery, while Llama-3.2 and Pixtral under 1D padding show high PVS with low AUSC.LLaVA-1.5 instead shows low PVS with moderate AUSC.Source: §5.2.2 · p. 8

6 Conclusion

• Only a small fraction of incorrect VQA predictions stem from fundamental reasoning limitations; many are associated with preprocessing information loss, encoder instability, or attention dilution.The authors conclude that addressing these errors calls for more robust visual encoders, adapters, and evaluation protocols that disentangle perception, tokenization, and reasoning effects.Source: §6 · p. 8

A Limitations

• The introduced metrics are intended for comparing models on the same image–question pairs and diagnosing instability, not as absolute numeric measures.Failure-mode attributions are not mutually exclusive; they identify the first likely point of information loss along the processing chain.Source: A Limitations · p. 11

2a. Patch Boundary Sensitivity

• For patch-boundary sensitivity, listed proposals include overlapping patches, continuous tokenization, and random patch-offset augmentation; for density mismatch, they include adaptive token allocation and detail-focused pretraining.The passages list these as possible interventions, not as experimentally validated remedies.Source: 2a. Patch Boundary Sensitivity · p. 11, 2b. Information Density Mismatch · p. 11

3. Attention Dilution

• For attention dilution and vision–language alignment, listed proposals include hierarchical or local attention and richer projection layers; position-aware attention and position-randomized training are also listed.The supplied recommendations additionally mention spatial-attention supervision and localization-aware tasks.Source: §3 · p. 11, §4 · p. 11, C Additional Analysis · p. 11

C.1 Is Padding an Out-of-Distribution Artifact?

• All models perform better when irrelevant content is replaced with white space, indicating that white padding itself is not the primary cause of degradation.The authors attribute the drop instead to reduced effective information density.Source: §C.1 · p. 12, Figure 5 · p. 12

C.2 Limitations of Single-Resolution VQA Accuracy

• Single-resolution accuracy cannot distinguish persistent failures from predictions that change under semantics-preserving transformations, motivating AUSC and PVS as complementary measures.AUSC tracks preservation of initially correct predictions, while PVS characterizes variation among initially incorrect predictions.Source: §C.2 · pp. 12–13
• For Llama-3.2-11B-Vision, 11.4% of samples remain incorrect under every tested condition, while 38.7% are correct under at least one alternative transformation.The results show that original errors can be recoverable and that originally correct predictions can also become incorrect after image-statistic changes.Source: §C.2 · p. 12, Figure 6 · p. 12

C.3 Comparison with Related Robustness Metrics

• ACE ranks Qwen2-VL-2B above Qwen3-VL-8B (0.078 versus 0.109), whereas AUSC reverses the ordering (0.793 versus 0.816).ACE reflects adjacent-resolution accuracy changes; AUSC measures retention of initially correct predictions across the resolution range.Source: §C.3 · p. 13, Table 4 · p. 13
• Flip− ranks Qwen3-VL-2B as more unstable than Llama-3.2-11B (0.085 versus 0.083), whereas PVS reverses the ordering (0.409 versus 0.477).Normalizing Flip− by initially incorrect samples also reverses the ordering, to 0.159 for Qwen3-VL-2B and 0.166 for Llama-3.2-11B.Source: §C.3 · p. 13, Table 4 · p. 13

C.4 Per-Benchmark Analysis of Resolution Sensitivity

• AUSC patterns vary by benchmark and model: many Qwen models show higher values on MMEval, MMStar, and MMVet than on CharXiv and MathVerse, but no benchmark is uniformly most stable.Transformation-specific curves also differ by model; interpolation is more damaging than padding for Qwen3-VL-8B on several benchmarks, while the reverse often holds for Pixtral-12B.Source: §C.4 · pp. 14–15
• High AUSC can coexist with high PVS: several models show both on MMEval, so retention of correct predictions and variability among initially incorrect predictions are distinct behaviors.Llama-3.2-11B-Vision and Pixtral-12B also show substantially higher MMEval PVS under tokenization-aligned 1D padding than under interpolation.Source: §C.4 · p. 15

C.5 Per Image Size Analysis

• For seven of ten models, PVS is lower on large than on small images, while AUSC varies only modestly across image-size strata; the PVS trend is architecture dependent.Qwen3-VL-2B and Llama-3.2-11B-Vision are among the exceptions to the general PVS pattern.Source: §C.5 · p. 15, Figure 9 · p. 15

D Human Review of Augmented Images

• Human annotators judged the vast majority of padded and interpolated images semantically equivalent to the originals; interpolation occasionally caused mild blur but rarely changed task-relevant information.Preliminary super-resolution experiments produced hallucinations and texture artifacts, especially in fine-grained regions, and were excluded from the final evaluation.Source: D Human Review of Augmented Images · p. 16

E.1 Area Under the Scaling Curve (AUSC)

• AUSC measures the area under the accuracy-versus-scale curve on Bcorrect, normalized to [0, 1].AUSC = 1 indicates perfect scaling robustness; lower values indicate sensitivity and faster degradation under transformations.Source: Figure 10 · p. 16, §E.1 · pp. 16–17
• AUSC depends on the size of Bcorrect; the authors recommend at least 75 originally correct examples for statistical significance.Source: §E.1 · p. 17

E.1.1 Prediction Variance Score (PVS)

• PVS can reveal sensitivity to tokenization or visual-encoding artifacts by showing whether resolution changes rescue incorrect predictions.Source: §E.1.1 · p. 17
• PVS measures how often a transformation makes an initially incorrect prediction correct, with values ranging from 0 to 1.PVS = 0 means incorrect predictions remain incorrect across resolutions; intermediate values indicate some predictions flip to correct.Source: §E.1.1 · p. 17
• Token boundary alignment affects predictions, while high PVS may signal brittleness in attention or reasoning pathways.The authors also note potential benefits from training with similar transformations.Source: §E.1.1 · p. 17

• GPT-5-nano receives the question, ground-truth answer, and model prediction, then returns a binary correctness judgment used as the final label for open-form matching.The prompt instructs the judge to score answers by whether they contain the same information as the ground truth.Source: F LLM-as-a-Judge Evaluation · pp. 17–18
• Over 97% agreement with human annotations was found when GPT-5-nano judgments were compared against 500 manually reviewed answer pairs.The sample spanned all benchmarks and transformation settings.Source: F LLM-as-a-Judge Evaluation · p. 18
• The evaluation is sensitive to the prompt used and the choice of judge model.Source: F LLM-as-a-Judge Evaluation · p. 18

• The experiments used 4×NVIDIA RTX A6000 GPUs, while a single-GPU setup is compatible but runs proportionally longer.Source: G Compute Resources · p. 18