PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Limits of Scale: Inverse Scaling in Vision-Language Models

Rohit Saxena, Alessandro Suglia, Marek Rei, Pasquale Minervini

arXiv:2610.27360v1cs.CVcs.AI

TL;DR

Whether vision-language models show inverse scaling on standard benchmarks, beyond constructed distractor settings, remains an open question, and aggregate scores can obscure subtask regressions. The authors evaluate model families using per-question pairwise analysis and find widespread inverse scaling across subtasks and families. Their results suggest that model selection should account for the target task.

  • Problem

    Whether inverse scaling occurs in VLMs under standard benchmark evaluation remains open, while aggregate reporting can obscure regressions in particular capabilities.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates 19 models across four families and uses per-question pairwise analysis to identify subtask-level inverse scaling.

    Source: §7 · p. 9

  • Results

    375 subtask–model-pair cases show inverse scaling, with 13 subtask types exhibiting the pattern across multiple families.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Model selection should account for downstream tasks rather than relying only on aggregate benchmark scores.

    Source: §6 · p. 9

    The study does not establish causal explanations for the observed inverse-scaling patterns.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures. We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific. The most affected domains in our study include multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases, while diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and suggest that model selection should be task-aware.

1 INTRODUCTION

This study asks whether inverse scaling occurs in vision-language models on standard benchmarks, where aggregate accuracy may conceal regressions in particular visual-reasoning categories. Across four model families, the authors find widespread subtask-level inverse scaling and argue for task-aware model selection.

1 INTRODUCTION

  • Provider reports already show smaller-model advantages on BLINK and VLMs-Are-Blind, and declining CountBench accuracy from Qwen3-VL 2B through 8B.Aggregate reporting lists these results without identifying which capabilities or questions regress.Source: §1 · p. 2
  • The study evaluates 19 models from four families and uses per-question pairwise flips to detect regressions hidden by aggregate accuracy.The analysis covers 29,386 questions per model and compares how often smaller and larger models succeed or fail on the same items.Source: §1 · p. 2
  • 375 subtask–model-pair cases show inverse scaling across all four families and every evaluated benchmark, with 13 subtask–dataset pairs recurring across multiple families.Multi-image spatial reasoning, visual understanding, and university-level STEM are strongly affected, while diagram understanding appears comparatively resistant.Source: §1 · p. 2
  • The findings challenge uniform-improvement assumptions and suggest choosing models by the capabilities required for the target application.The paper’s qualitative examples include misidentifying visible objects and selecting plausible but incorrect options.Source: §1 · p. 2

2 RELATED WORK

Prior scaling work largely studies aggregate performance, while inverse-scaling research has identified failures on selected language and reasoning tasks. This paper examines subtask-level scaling in open VLM families using standard multimodal benchmarks and per-question comparisons.

Scaling laws

  • Earlier scaling studies report aggregate regularities, while this work asks whether improvements persist across finer-grained benchmark subtasks within a VLM family.The authors assume these subtasks correlate more directly with specific multimodal skills.Source: §2 · p. 3

Inverse scaling in LLMs and VLMs

  • Prior work finds inverse scaling or non-monotonic behavior in language and reasoning models, and visual distractors can reduce VLM accuracy in constructed settings.Wei et al. (2023) report U-shaped curves, while Bae et al. (2025) study visual distractors in multimodal reasoning.Source: §2 · p. 3
  • This study examines open VLM families on standard benchmarks and detects inverse scaling through per-question pairwise flips within existing subtasks.The authors characterize their findings as observational evidence from publicly available checkpoints.Source: §2 · p. 3

3 METHODOLOGY: INVESTIGATING INVERSE SCALING IN VLMS

The methodology defines inverse scaling as a smaller model outperforming a larger model on a benchmark subtask, then measures paired question-level disagreements across models within each family.

Problem Formulation

  • A subtask exhibits inverse scaling for a model pair when the smaller model’s accuracy exceeds the larger model’s accuracy.Models are ordered by total parameter count; dense and MoE variants can differ in active parameters.Source: §3 · p. 3

Per-Question Flip Analysis

  • Per-question flip analysis compares correctness on shared subtask questions, distinguishing disagreements favoring the smaller model from those favoring the larger model.Net equals FlipInv minus FlipNorm; positive Net means the smaller model is correct more often among discordant questions.Source: §3 · p. 3

Models and benchmarks

  • The evaluation covers 19 models across four open-weight families, including dense and Mixture-of-Experts variants.Table 1 summarizes the model families and benchmarks, with MoE active-parameter counts shown in parentheses.Source: Table 1 · p. 4

4 EXPERIMENTAL SETUP

The experiments evaluate diverse multimodal benchmarks and model families, then apply a zero-shot protocol and explicit thresholds to select inverse-scaling cases.

4 EXPERIMENTAL SETUP

  • The setup evaluates 19 models from four open-weight families and nine benchmark splits spanning perception, diagrams, science, multi-image vision, multimodal reasoning, university knowledge, and mathematics.The benchmark suite yields 29,386 unique questions per model and 558,334 model–question evaluations.Source: §4 · p. 4

4.1 EVALUATION PROTOCOL

  • Models are evaluated zero-shot with direct prompts and greedy decoding, with thinking modes disabled and deterministic or numeric answer matching where possible.A multimodal fallback adjudicator is used when verification rules are inconclusive; it handles 0.4% of responses.Source: §4.1 · p. 4
  • A reported inverse-scaling case requires at least 20 shared questions and Net ≥5; stricter thresholds leave 132 cases across 44 subtask–dataset pairs.The stricter threshold retains four cross-family subtasks.Source: §4.1 · p. 4

5 RESULTS

Aggregate performance mostly scales monotonically, but subtask-level inverse scaling appears across all four model families and every evaluated benchmark. The results span recurring cross-family subtasks, non-monotonic size trajectories, and sampled flips that are mostly judged genuine errors.

5.1 AGGREGATE SCALING BEHAVIOR

  • The largest model usually has the highest or near-highest aggregate accuracy, but Molmo 4B exceeds 8B on BLINK and MMBench.On BLINK, Qwen 30B-A3B scores 62.4%, below Qwen 4B at 66.4% and Molmo 4B at 63.5%; Molmo 4B scores 88.3% versus 87.9% for 8B on MMBench.Source: §5.1 · pp. 4–5

5.2 SUBTASK-LEVEL INVERSE SCALING

  • Inverse scaling appears across all four model families and every evaluated benchmark, despite mostly monotonic aggregate performance.The 375 cases span 115 subtask–dataset pairs and 171 family–subtask combinations; 87% of family–subtask pairs have non-monotonic accuracy curves.Source: §5.2 · p. 5

5.3 PER-FAMILY ANALYSIS

  • Inverse scaling is reported across Qwen3-VL, InternVL3.5, Gemma 3, and Molmo 2, with smaller models outperforming larger counterparts on cited BLINK and ScienceQA comparisons.On BLINK Jigsaw, Qwen3-VL 4B scores 84.0% versus 60.0% for 30B-A3B; on ScienceQA Particle motion and energy, Gemma 3 12B scores 95.1% versus 51.2% for 27B; on BLINK Jigsaw, Molmo 2 4B scores 79.3% versus 58.7% for 8B.Source: §5.3 · p. 6

5.4 CROSS-FAMILY PATTERNS

  • BLINK multi-image spatial reasoning is the clearest cross-family pattern: five of 14 subtasks show replication, and Multi-view Reasoning appears in all four families.Its cross-family cases reach Max Net values up to 36; cross-family replications also include coarse perception, university-level MMMU STEM subjects, autonomous driving, and MME Scene.Source: §5.4 · p. 6

5.5 OTHER FINDINGS

  • MoE models often appear as the larger model in inverse-scaling pairs, but active parameter count, training recipe, and alignment procedure confound attribution to expert routing.The dense-only analysis still finds 193 cases, so inverse scaling is not solely an MoE artifact.Source: §5.5 · pp. 7–8
  • Size trajectories can be non-monotonic, and the smallest model sometimes leads: Qwen 2B scores 65.0% on MMMU AI Content Detection versus 45.0% for 235B.The twofold-size analysis retains cases but finds smaller-model leads of Net ≥5 in only 9 of 666 largest-gap comparisons, so such cases are limited across the full size range.Source: §5.5 · pp. 7–8
  • Inverse scaling is not confined to the largest-size comparisons: 326 cases remain when the larger model has at least twice the total parameters, and 193 remain among dense-only pairs.Both twofold-size restrictions retain 11 of 13 cross-family subtasks; excluding MoE variants still leaves eight subtasks replicated across at least three families, including BLINK Multi-view Reasoning.Source: §5.5 · p. 8

5.6 ERROR TAXONOMY

  • Human reviewers judged at least 84% of sampled inverse-scaling flips to reflect real model failures, while flagging 15.7% as annotation problems or ambiguous.On visual-reasoning benchmarks BLINK and ScienceQA, humans and LLM judges agree that nearly all flips are genuine errors.Source: §5.6 · pp. 8–9

6 DISCUSSION

The paper discusses possible explanations for inverse scaling, tests whether chain-of-thought prompting reduces it, and considers implications for model selection. It also notes limits on causal interpretation and scope.

Why does inverse scaling occur in VLMs?

  • The authors offer four hypotheses—localized capability regression, distractor sensitivity, MoE-specific factors, and training-data composition—but do not establish causal explanations.Their observational data cannot attribute regressions to parameter count alone or separate MoE routing, active parameters, and training differences.Source: §6 · p. 9

Can chain-of-thought prompting recover the errors?

  • CoT reduced the median Net from +8 to 0, but 96 of 375 cases (25.6%) retained Net ≥5 and the larger model recovered 43.5% of questions it had lost.The smaller model’s accuracy also decreased under CoT in 251 of 375 cases, so part of the Net reduction reflects lost smaller-model accuracy; whether recovery reflects genuine reasoning improvement remains open.Source: §6 · p. 9

Implications for model selection

  • On a minority of evaluated comparisons, smaller models outperform larger ones, especially in multi-image spatial reasoning, visual perception, and specific STEM subtasks.The authors recommend evaluating candidate models on downstream tasks rather than aggregate scores.Source: §6 · p. 9

Limitations

  • The study evaluates only open-weight models, weights questions equally, and cannot isolate parameter-count effects or rule out size-dependent benchmark contamination.Option order affects results, and paraphrases are untested.Source: §6 · p. 9

7 CONCLUSION

The paper concludes that VLM performance can scale non-monotonically across subtasks, with inverse scaling observed across model families. The findings motivate research on mechanisms and task-aware deployment.

7 CONCLUSION

  • Across four model families and eight benchmarks, per-question pairwise analysis identified 375 inverse-scaling cases, with 13 subtask types showing the pattern across multiple families.The most affected areas were multi-image spatial reasoning, visual perception, and specialized STEM knowledge.Source: §7 · p. 9
  • The findings motivate work on inverse-scaling mechanisms and suggest task-aware model deployment.The paper also calls for robust multimodal models that leverage their full capacity across modalities.Source: §7 · p. 10

A ADDITIONAL EXPERIMENT DETAILS

The appendices document evaluated models, inference settings, and answer verification, then provide aggregate accuracy and full pairwise inverse-scaling results across model families.

A ADDITIONAL EXPERIMENT DETAILS

  • All 19 evaluated models are open-weight, and Table 5 lists their exact HuggingFace identifiers.Inference used NVIDIA H100 GPUs without quantization; the evaluation comprised 171 runs and 558,334 model-question evaluations.Source: A ADDITIONAL EXPERIMENT DETAILS · p. 13, Table 5 · p. 14
  • Models used zero-shot direct prompting with greedy decoding and disabled thinking modes; GPT-5.4 adjudicated 0.4% of responses when deterministic verification was inconclusive.The appendices provide prompt settings and answer-verification details.Source: A ADDITIONAL EXPERIMENT DETAILS · p. 14

B AGGREGATE ACCURACY TABLES

  • The aggregate accuracy tables report overall accuracy (%) for Qwen3-VL, InternVL3.5, Gemma 3, and Molmo 2, marking the best result in each family.For Gemma 3 and Molmo 2, underlining marks when the largest model is not best.Source: Table 6 · p. 14, Table 7 · p. 15, Table 8 · p. 15

C FULL PAIRWISE INVERSE SCALING RESULTS

  • The full pairwise results list cases using Net ≥5 on at least 20 matched questions, with Net defined as FlipInv minus FlipNorm.FlipInv counts questions where the smaller model is correct and the larger is wrong; FlipNorm counts the reverse.Source: C FULL PAIRWISE INVERSE SCALING RESULTS · p. 15

C.1 QWEN3-VL (122 CASES)

  • Table 9 presents dense-only cross-family replication results for subtask–dataset pairs with inverse scaling in at least three of the four model families, based solely on dense model pairs.Source: Table 9 · p. 17

D STATISTICAL SIGNIFICANCE ANALYSIS

The paper characterizes paired asymmetry with exploratory significance tests, reports cross-family replication and threshold robustness, and documents dataset-specific prompt formats.

D STATISTICAL SIGNIFICANCE ANALYSIS

  • The one-sided McNemar exact tests characterize paired asymmetry across 375 cases, but their p-values are exploratory rather than confirmatory because cases were preselected and comparisons overlap.The authors also report Benjamini–Hochberg adjusted p-values over reported cases and all 10,434 eligible comparisons.Source: D STATISTICAL SIGNIFICANCE ANALYSIS · p. 24, Table 35 · p. 25

E CROSS-FAMILY REPLICATION

F NET FLIP DISTRIBUTION

  • The same benchmarks and subtasks dominate across different Net thresholds, indicating that the qualitative pattern is stable to threshold choice.Table 38 summarizes case counts, unique subtask-dataset pairs, and replicated subtasks at different thresholds.Source: F NET FLIP DISTRIBUTION · p. 25, Table 38 · p. 26

G EVALUATION PROMPTS

  • Evaluation prompts were dataset-specific: multiple-choice tasks requested an answer letter, while open-ended tasks requested a concise answer or final value.MME was presented as-is; AI2D, MathVista, and BLINK used pre-formatted prompts.Source: G EVALUATION PROMPTS · pp. 26–27

G.1 ERROR TAXONOMY JUDGE

The appendix checks how inverse-scaling findings depend on annotation, comparison size, option order, and vision-encoder differences. It reports that cross-family replication persists under several restrictions, while some case counts and magnitudes vary.

G.1 ERROR TAXONOMY JUDGE

  • The error taxonomy uses blinded multimodal judges to assign one failure category per question, and the authors treat these labels as exploratory rather than validated ground truth.GPT-5.4 and Claude Opus 4.6 agree on 90.8% of questions (Cohen’s κ = 0.30).Source: §G.1 · pp. 27–28

H HUMAN ANNOTATION VALIDATION

  • Human raters agree with LLM judges on most sampled flips, but classify more as annotation problems; all five raters judge the large majority as genuine model errors.Human majority vote agrees with GPT-5.4 on 88.2% and Claude Opus 4.6 on 84.3%; humans label 15.7% as bad annotation or ambiguous, versus 2.6% and 10.5%.Source: H HUMAN ANNOTATION VALIDATION · p. 28

I RESTRICTING TO COMPARISONS OF AT LEAST TWOFOLD SIZE

  • 326 of 375 cases remain under the total-parameter restriction, while 239 remain under the active-parameter restriction; 11 of 13 replicated subtasks survive both.The excluded active-parameter comparisons had a 6.1% case rate and no replicated subtasks; BLINK falls from 95 to 58 cases and MathVista to none.Source: I RESTRICTING TO COMPARISONS OF AT… · pp. 28–29

J OPTION-ORDER SENSITIVITY

  • The smaller model’s net advantage stays positive in all 17 option-order and identity-replay runs, although reordering changes its magnitude.For Qwen 4B vs. 32B on Jigsaw, reversal reduces Net from +30 to +9; the identity replays vary by up to 11 Net points.Source: J OPTION-ORDER SENSITIVITY · pp. 29–30

K VISION ENCODERS AND IMAGE PREPROCESSING

  • 271 of 375 cases compare models with the same encoder tier, and 10 of 13 replicated subtasks remain when analysis is restricted to same-tier cases.This includes BLINK Multi-view Reasoning in all four families.Source: K VISION ENCODERS AND IMAGE… · p. 30

L CHAIN-OF-THOUGHT CONTROL EXPERIMENT

The control re-runs both models in each inverse-scaling case with chain-of-thought prompting. This attenuates inverse scaling overall, but its persistence varies across benchmarks and many cases remain significant.

Result

  • The median Net falls from +8 to 0 under chain-of-thought prompting, but 96 of 375 cases (25.6%) retain Net ≥5 and 101 reverse sign.The larger model recovers 3,640 of 8,361 questions (43.5%) lost under direct prompting, while the smaller model’s accuracy decreases in 251 cases (mean −3.9pp).Source: L CHAIN-OF-THOUGHT CONTROL EXPERIMENT · p. 31

Where it persists

  • Persistence varies by benchmark: 48.6% of MME and 42.1% of BLINK cases survive, compared with 7.6% of MMMU test-split cases.The authors suggest, without directly testing it, that chain-of-thought may help more with reasoning over recognized content than with extracting answers from images.Source: L CHAIN-OF-THOUGHT CONTROL EXPERIMENT · p. 31

Significance

  • Of 96 cases persisting under chain-of-thought, 18 remain significant after BH correction over 375 tests, compared with 54 of the same cases under direct prompting.44 persisting cases have uncorrected p < 0.05.Source: L CHAIN-OF-THOUGHT CONTROL EXPERIMENT · p. 31

M QUALITATIVE EXAMPLES OF INVERSE SCALING

The examples show smaller models answering correctly where larger same-family models fail across visual perception, spatial reasoning, and knowledge tasks. Some larger-model errors involve detailed reasoning built on a mistaken visual premise.

BLINK / Jigsaw

BLINK / Multi-view Reasoning

MME / Celebrity

  • In MME Celebrity, the larger InternVL model fails to identify Arnold Schwarzenegger, while the smaller model answers correctly.The analysis describes this as a straightforward visual identification failure.Source: M QUALITATIVE EXAMPLES OF INVERSE… · p. 33

MathVista / Geometry

ScienceQA / Astronomy

BLINK / Forensic Detection

  • On BLINK Forensic Detection, smaller Qwen models identify the real photograph with and without chain-of-thought, while the largest rejects it under both prompts.The largest model reasons through the images but judges the real photograph suspicious for AI generation.Source: M QUALITATIVE EXAMPLES OF INVERSE… · p. 36

MME / Landmark

  • On MME Landmark, InternVL 1B identifies the Church of Our Lady of Sorrows in Riga, while 2B rejects the correct landmark and justifies its rejection at length.This pair is among the ten strongest cases.Source: M QUALITATIVE EXAMPLES OF INVERSE… · p. 37

Related papers

Loading 2610.27360v1…