PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models

Sanghee Park, Kee-Eung Kim

arXiv:2609.27395v1cs.CL

TL;DR

Compact-VLM benchmarks can saturate or compress models into single accuracy scores, leaving behavioral differences unmeasured. PRISM-VLM evaluates seven axes on items recycled from public benchmarks and aggregates them into PSCORE. PSCORE separates more model pairs than prior single benchmarks, while per-axis profiles reveal differences, especially in sycophancy resistance.

  • Problem

    Compact-VLM benchmarks reduce models to single accuracy scores that can saturate or compress model differences and do not surface behavioral distinctions.

    Source: §1 · p. 1

  • Method

    PRISM-VLM scores compact VLMs on seven axes using recycled public-benchmark items and combines the axis scores into PSCORE.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    97% of 861 model pairs are separated by PSCORE under an item-level paired bootstrap, versus 44–80% for any single prior benchmark.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Per-axis profiles reveal behavioral differences, particularly sycophancy resistance, which is nearly orthogonal to single-prompt accuracy.

    Source: §1 · p. 1, §6 · p. 9

    The evaluation covers 42 compact models on English image-and-text items and leaves calibration, tool use, long-horizon dialogue, and multilingual transfer uncovered.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Compact vision-language models (VLMs) now power a growing share of multimodal applications. The benchmarks used to compare them, however, inherit a frontier-centric design: each model is reduced to a single accuracy number, narrowing the inter-model gap on saturated suites and pressing models into low-score bands on harder ones. We introduce PRISM-VLM, a multi-axis discriminative benchmark that scores every item along seven axes covering the recurring failure modes (task quality, behavioral robustness, and capability bottlenecks) and combines them into a single PScore, with items recycled from fifteen public benchmarks. Across compact VLMs from the past two years, PScore separates model pairs more reliably than prior single-axis benchmarks under an item-level paired bootstrap, and surfaces behavioral differences these benchmarks average away. Even models with statistically indistinguishable PScores diverge sharply along the per-axis profile, particularly on sycophancy, which is nearly orthogonal to single-prompt accuracy. We will release the full pipeline, prompts, and per-item annotations.

1 Introduction

Existing compact-VLM benchmarks either saturate or compress model differences into single accuracy scores, missing behavioral and capability distinctions relevant to deployment. PRISM-VLM evaluates seven axes and combines them into PSCORE, which more reliably separates models than prior single-benchmark scores.

1 Introduction

  • Single-accuracy benchmarks saturate on compact models or compress their scores, while omitting behavioral differences such as hallucination, sycophancy, and instruction following.The paper identifies behavioral robustness and capability bottlenecks as distinct clusters of model differences.Source: §1 · p. 1
  • PRISM-VLM evaluates seven recurring failure-mode axes, retaining axes that add discriminative signal beyond the others and can be measured on public items.Resistance to conversational pressure is nearly orthogonal to single-prompt accuracy; hallucination resistance and instruction following are only partially orthogonal.Source: §1 · p. 1
  • The benchmark adds sycophancy resistance as a near-orthogonal behavioral probe and treats multi-question load as an amplifier of inter-model spread.The axes also include hallucination, instruction following, counting, and visual discrimination.Source: §1 · p. 2
  • PSCORE combines the seven axes by harmonic mean, and each axis is admitted only when independent judges achieve Spearman ρ ≥0.95.The harmonic mean is less recoverable from task quality alone than arithmetic or geometric means.Source: §1 · p. 2
  • 97% of 861 model pairs are separated by PSCORE under an item-level paired bootstrap, versus 44–80% for any single prior benchmark.The authors will release the full pipeline, prompts, and per-item annotations.Source: §1 · p. 2

2 Related Work

Prior VLM benchmarks commonly collapse performance across tasks into one accuracy score, while compact-tier suites often measure efficiency rather than behavioral differences. PRISM-VLM combines behavioral and capability axes to address saturation and examine cross-axis patterns.

2 Related Work

  • Established VQA suites have saturated on recent compact models, while harder frontier-targeted benchmarks raise difficulty without necessarily improving separation among comparable models.Holistic suites such as MMMU, MMBench, MME, and MMStar aggregate tasks into one accuracy number.Source: §2 · p. 2
  • PRISM-VLM is presented as the first multi-axis discriminative benchmark designed specifically for compact VLMs; existing compact-targeted suites focus on throughput or memory.Source: §2 · p. 3
  • Earlier work studied most behavioral failure modes in isolation, so PRISM-VLM’s integration exposes cross-axis patterns such as sycophancy resistance’s near-orthogonality to single-prompt accuracy.Multi-question robustness had not, to the authors’ knowledge, been used as a VLM axis.Source: §2 · p. 3

3 PRISM-VLM

PRISM-VLM recycles public benchmark items and adds per-item perturbations to measure seven axes of compact-VLM behavior. Human verification, axis-specific grading, and a harmonic-mean aggregate provide a reproducible evaluation pipeline.

3 PRISM-VLM

  • Seven per-axis means are combined into PSCORE by harmonic mean, so strong performance on one axis cannot mask weakness on another.The reported pool uses all seven axes surviving the cross-judge audit.Source: §3 · pp. 3–4
  • The evaluation draws a deduplicated union of 6,238 items from 15 public benchmarks using five deterministic seeds, with each model evaluated on every applicable item.The five seeds yield 401–496 unique items for 13 benchmarks; two smaller upstream pools cap at 205 and 120 items.Source: §3 · p. 3
  • Per-item perturbations are synthesized for hallucination resistance, multi-question load, counting, and visual discrimination, with every generated artifact manually reviewed for validity.The four judge-built axes contain 24,952 entries in total; the same seeds fix the item set across models.Source: §3 · p. 4
  • The seven axes use dedicated judge prompts for open-ended grading and deterministic matching for closed-answer axes; each axis score is its applicable-item mean.The pool has applicable items for all seven axes, with no items dropped for missing build-time counting golds.Source: §3 · p. 4
  • Instruction following appends one to three verifiable formatting constraints and scores the mean constraint compliance, making longer constraint sets harder.The constraint pool covers length, casing, keywords, output format, and answer shape.Source: §3 · p. 4
  • Hallucination resistance tests trap questions about absent entities, misleading premises, or uncertain images, rewarding refusal or premise correction over hallucination.The three trap subtypes are distributed across approximately equal portions of the items.Source: §3 · p. 4
  • Multi-question robustness bundles two to four added questions with the original and scores an item correct only when every answer is correct.Sycophancy resistance tests a three-turn pressure dialogue only on items answered correctly at turn zero, scoring stability across all turns.Source: §3 · p. 4
  • The harmonic mean is lower-bounded by the geometric and arithmetic means and gives near-zero axes a stronger pull on PSCORE.All axis means are positive, with a minimum of 0.011, so the harmonic mean is defined without epsilon smoothing.Source: §3 · p. 5

4 Experimental Setup

The experiments evaluate a diverse set of compact models under controlled, low-effort inference settings. Fixed decoding and released item-level judgments are intended to make reported results reproducible.

4 Experimental Setup

  • The evaluation includes 42 compact models released from March 2024 through April 2026: 17 closed-API models from five providers and 25 open-weight models.Provider diversity was prioritized over volume.Source: §4 · p. 5
  • Models run at their lowest exposed reasoning effort, with temperature 0 and per-axis output-token budgets.Extended reasoning was excluded to avoid confounding model discrimination with provider-specific reasoning-effort differences.Source: §4 · p. 5
  • Each model is evaluated once on the 6,238-item set, and deterministic decoding makes per-axis means and PSCORE reproducible for a fixed model snapshot.Per-item judgments and model responses are slated for release with the evaluation pipeline after internal review.Source: §4 · p. 5

5 Results and Analysis

PRISM-VLM’s PSCORE statistically separates more compact-model pairs than prior benchmarks, while per-axis results reveal behavioral differences that the aggregate can conceal. The analysis also tests the aggregate’s sensitivity to axis choice and judge variation.

5.1 Main leaderboard

  • ∼60 pp versus ∼18 pp: PSCORE’s closed-API score range is ∼241% wider than the three narrowest-range prior benchmarks.The widening persists among the top five closed-API models, where it is +131%.Source: §5.1 · pp. 6–7, Figure 2 · p. 7
  • 97.0% of 861 model pairs are separated by PSCORE, versus 44–80% for any prior benchmark and 88.7% for their macro mean.At matched item budgets, PSCORE still separates 81–88% of pairs.Source: §5.1 · p. 6, Table 3 · p. 7
  • Models within a 0.007 PSCORE window have distinct axis strengths: sycophancy favors Qwen3-VL-2B, while other axes favor Gemini 2.0 Flash or GPT-4.1 mini.The three models are statistically indistinguishable; no pairwise difference excludes zero under the reported bootstrap.Source: §5.1 · p. 7, Table 5 · p. 7

5.2 Per-axis behavior and inter-axis correlations

  • Across models, sycophancy resistance is the most orthogonal axis, with |ρ|≤0.26 to every other axis; q and mq instead correlate at ρ=0.95.Despite that correlation, mq’s inter-model range is 0.494 versus 0.323 for q.Source: §5.2 · pp. 7–8, Figure 3 · p. 8
  • The seven axes are correlated diagnostic probes, not independent dimensions, and PSCORE is a chosen bottleneck-sensitive summary rather than a uniquely principled aggregate.The harmonic mean is bottlenecked by mq for 26 of 42 models, sc for 14, and h for two.Source: §5.2 · p. 8
  • 20.0% of turn-0-correct items flip at Q1, and only 64.3% survive to the Q3 hint; 86% of Q3 flips adopt its suggested answer.Correcting the sycophancy axis leaves the ranking stable (ρ=0.88) and pairwise separation at 95.9%.Source: §5.2 · pp. 8–9

5.3 PSCORE validity and judge audit

  • Harmonic aggregation is least recoverable from q alone (r=+0.79, R2=0.62, versus +0.87 for arithmetic); dropping sc lowers rank correlation with PSCORE to 0.89.Other aggregation choices and axis reweightings retain Spearman ρ≥0.94 against PSCORE.Source: §5.3 · p. 9, Table 6 · p. 9
  • The independent-judge audit yields overall ρ=0.99 across model rankings, with every axis above the 0.95 admission threshold and a minimum of 0.96 on c.For seven OpenAI evaluees, the maximum rank shift is two positions; the authors note the per-family sample is modest.Source: Table 7 · p. 9, §5.3 · p. 9

6 Conclusion

PRISM-VLM is a seven-axis discriminative benchmark for compact VLMs, aggregated by harmonic mean into PSCORE. Across 42 models, it separates more model pairs than prior benchmarks and highlights conversational-pressure robustness as a distinctive axis.

6 Conclusion

  • PSCORE separates 97% of model pairs under an item-level paired bootstrap, versus 44–80% for any prior benchmark and 88.7% for their macro mean.The separation gap is not memorization-driven and holds under a cross-judge audit.Source: §6 · p. 9
  • The sycophancy axis has |ρ| ≤0.26 with every other axis and is unrecoverable from existing benchmarks.Separating adoption of a planted false answer from generic pressure-instability keeps the axis near-orthogonal (|ρ|≤0.13) and the ranking stable (ρ=0.88).Source: §6 · p. 9
  • The authors conclude that compact-tier progress should be measured along behavioral axes, particularly sycophancy resistance, alongside raw accuracy.PRISM-VLM combines seven axes by harmonic mean into PSCORE.Source: §6 · p. 9

Limitations

PRISM-VLM reports statistical separation among compact VLMs, but its scope and validation do not establish external validity or exhaustive behavioral coverage.

Limitations

  • The evaluation covers 42 compact checkpoints on English image-and-text items; calibration, tool use, long-horizon dialogue, and multilingual transfer remain uncovered.The pipeline can accommodate additional axes without retiring existing ones.Source: Limitations · p. 10
  • The paper claims statistical separation, not that PSCORE predicts downstream task success, human preference, or incident rates.The authors identify relating released per-axis profiles to independent signals as a next step.Source: Limitations · p. 10
  • A cross-judge audit confirms rank stability but cannot rule out failure modes shared by both judges or by judges and human annotators.Both Jgen and most evaluated models come from U.S. frontier providers, leaving common ecosystem biases unresolved.Source: Limitations · p. 10
  • The sycophancy measure brackets genuine sycophancy because dialogue stops at the first flip, so early-folding models are not shown a false answer.The authors say running all three turns regardless of flips would close this gap.Source: Limitations · p. 10

Ethical Considerations

PRISM-VLM reuses public benchmark items and reports its release plans and annotation practices. The evaluation also has stated privacy, bias, and dual-use considerations.

Ethical Considerations

  • The pipeline code is planned for release under Apache-2.0 after internal review, while prompts, per-item annotations, and annotator instructions are under CC BY 4.0.Source benchmarks are used under their original public licenses.Source: Ethical Considerations · p. 10
  • The study reuses public items, releases no identifying information, and uses anonymized responses graded by in-house annotators without collecting personally identifying information.Annotators were salaried in-house researchers; no crowdworkers or per-task compensation were involved.Source: Ethical Considerations · p. 10
  • The English-only evaluation is not a comparative claim about multilingual or non-Western use, and the judge audit does not eliminate biases shared across the frontier-LLM ecosystem.Both Jgen and most evaluees come from U.S. frontier providers.Source: Ethical Considerations · p. 10
  • Adversarial hallucination and sycophancy templates are evaluation prompts only, and the scores measure resistance rather than compliance.The authors note that exposing specific failure modes is dual-use, while documenting them openly helps deployers more than attackers.Source: Ethical Considerations · p. 10

A Benchmarks

The benchmark draws reproducible item samples from fifteen public benchmarks and runs axes through synthesis, inference, and grading. The appendix describes axis-specific implementations, evaluation settings, and prompts.

A Benchmarks

  • Using five independent conventional seeds yielded a deduplicated pool of 6,238 evaluated rows, including 25 shared source items with independently synthesized perturbations.The pool contains 6,213 distinct source items; the five-seed union makes benchmark selection reproducible and less dependent on any single draw.Source: A Benchmarks · pp. 13–14

B Seven Axes

  • Each axis uses synthesis, inference, and grading; synthesis is trivial for q, if, and c, while the other four axes use Jgen to create per-item probes.For instruction-following, each item samples 1–3 constraints from an eleven-constraint pool, and the score averages compliance across them.Source: B Seven Axes · p. 14

C Parameters

  • After retries, 26 of 42 models answered all items, while the largest per-model residual was 233 of 6,238 (3.7%), concentrated on multi-image benchmarks for three open-weight models.Unscorable items are excluded rather than scored zero, avoiding conflation of input support with model ability; denominators therefore vary by this residual.Source: C Parameters · p. 14

D Prompts

  • Generation prompts create per-item perturbations at build time, grading prompts score responses at evaluation time, and sycophancy pressure turns draw from fixed template pools.Prompt variables such as {question} and {gold} are substituted per item.Source: D Prompts · p. 14, B Seven Axes · p. 14

E Robustness Analyses

The robustness checks test whether PRISM-VLM’s separation persists among strong models, under alternative scoring choices, and across model pools. Effective axis weights reflect the current pool’s score spread rather than fixed emphasis.

E Robustness Analyses

  • PSCORE’s range is about 2.3× the mean range of the three narrowest prior benchmarks, and the widening persists among top-K closed-API models.At K=10, widening drops to +54% as strong sycophancy-scoring closed-API models enter the subset.Source: E Robustness Analyses · pp. 14–16, Table 12 · p. 21
  • Dropping sycophancy causes the largest ranking shift (ρ=0.89), while dropping multi-question load, question answering, counting, or visual discrimination leaves rankings nearly unchanged.The sc axis carries the most signal beyond single-prompt accuracy.Source: E Robustness Analyses · p. 17, Table 13 · p. 21
  • Sycophancy is reliably the most orthogonal axis: its correlations with q, c, and vis are low, while its correlation with PSCORE is +0.62.The cited 95% bootstrap intervals for sc correlations with q, c, and vis include zero or remain below +0.5.Source: E Robustness Analyses · pp. 19, 21
  • Pairwise PSCORE separation replicates at 96.5% across 20 disjoint half-pools, compared with 97.0% for the full 42-model pool.Sycophancy remains the least-correlated axis in 16 of 20 halves and never ranks below second.Source: E Robustness Analyses · p. 21
  • The leaderboard is stable across near-tie, denominator, memorization-prone-axis, and partial-credit checks: the reported rank correlations are p=0.95/0.10, ρ=0.96, ρ=0.99, and ρ=0.98, respectively.The near-tie p-values concern the three models in the 0.007 PSCORE window; the other correlations compare rankings under the tested scoring alternatives.Source: E Robustness Analyses · p. 22
  • Judge verdicts agree with human consensus at overall κ=0.97, with per-axis κ from 0.93 to 1.00 and raw disagreement from 0% to 3.0%.The reported overall raw disagreement is 0.9%.Source: E Robustness Analyses · pp. 22–23

E.1 Effective axis weights

  • Across 42 models, multi-question load and sycophancy carry the highest mean effective weights (0.29 and 0.26), and are the dominant lowest-scoring axes for 26 and 14 models.The harmonic mean concentrates weight on each model’s lowest-scoring axes.Source: §E.1 · p. 23, Table 15 · p. 23
  • Effective weights depend on pool spread: sycophancy and visual discrimination span the widest ranges (0.879 and 0.868), while q spans 0.323; tested reweightings largely preserve rankings.Doubling hallucination and instruction-following gives ρ=0.99 against the default ranking with the same top four; alternative aggregators retain the same leader.Source: §E.1 · p. 23, Table 16 · p. 23

E.2 Sycophancy vs. generic pressure-instability

The leaderboard’s sycophancy score combines early pressure-induced answer changes with adoption of a false answer at the final turn. Decomposing these behaviors yields bounds that remain far apart for fragile models, while substituting a corrected score preserves strong benchmark discrimination.

E.2 Sycophancy vs. generic pressure-instability

  • Only 64.3% of turn-0-correct items reach the Q3 false hint: 20.0% flip at Q1, 15.8% at Q2, and 17.7% at Q3.Among Q3 flips, 24,584 of 28,590 (86%) are confirmed adoptions of the suggested false answer.Source: §E.2 · pp. 23–24
  • The sycophancy bounds nearly close for robust models but remain wide for fragile ones: Qwen3-VL-2B scores 0.990 versus 0.988, while Gemma-3-4B scores 0.879 versus 0.116.Gemma-3-4B carries 14% of its turn-0-correct items to Q3, compared with 90% for Qwen3-VL-2B; early-terminating dialogue cannot fully separate instability from sycophancy.Source: §E.2 · p. 24, Table 18 · p. 25
  • Replacing scB with sccorr preserves 95.9% pairwise separation versus 97.3% under scB, far above prior benchmarks’ 44–80%.The corrected axis is more orthogonal to the other axes than scB (maximum absolute Pearson correlation 0.13 versus 0.26).Source: §E.2 · pp. 24–25

F Robustness to Perturbation Generator and Judge Family

Robustness checks vary the perturbation-generating and grading judge families, then compare LLM judgments with human consensus. The LLM leaderboard is stable across judge checks, while the human subset supports broad ordering but does not resolve the top pair.

F Robustness to Perturbation Generator and Judge Family

  • Three paired generator-and-judge family cells produce the same leaderboard ordering, although absolute PSCORE values shift with judge strictness.The largest reported in-family shift is +0.100 for Gemini 3 Flash, still short of its 0.101 lead over Qwen3-VL-8B; the Claude cell grades Claude Haiku 4.5 more strictly (−0.044).Source: F Robustness to Perturbation Generator… · p. 25, Table 19 · p. 25
  • With Jgen fixed, swapping the grading judge preserves the leaderboard (ρ=0.99), with a maximum per-evaluee shift of |∆|=0.046.This check uses 200 items per axis and 1,400 items per evaluee.Source: F Robustness to Perturbation Generator… · p. 25, Table 20 · p. 26
  • Human and LLM judgments agree on the weakest evaluee and broad ordering, but cannot statistically separate Gemini 3 Flash and GPT-5 mini in the 200-item human subset.The human PSCORE difference is −0.016 for Gemini 3 Flash relative to GPT-5 mini, with a 95% CI of [−0.234, +0.148].Source: F Robustness to Perturbation Generator… · pp. 25–26, Table 21 · p. 26

G Worked Samples

Three worked samples trace individual items through PRISM-VLM’s seven axes, showing contrasting model behaviors and how per-item bookkeeping differs from the leaderboard’s scB denominator.

G Worked Samples

  • Each sample follows one source item through all seven axes, showing applicable perturbations, the model response, judge verdict, and rationale.The figures cover Gemini 3 Flash and Claude Haiku 4.5 on OK-VQA items, and Gemma-4-E4B on an AI2D food-web diagram.Source: G Worked Samples · p. 26, Figure 20 · p. 27, Figure 21 · p. 28
  • The examples contrast a sycophancy flip, Claude-family instability before any false answer is offered, and rigid agreement with a wrong committed answer.The Gemini sample flips at Q1 on sc; Claude Haiku 4.5 flips at Q1 before a false answer is offered; Gemma-4-E4B holds sc by rigid agreement with its wrong answer at Q.Source: G Worked Samples · p. 26
  • Per-item sc bookkeeping and leaderboard scB means use different denominators: scB includes only items whose q=1 at turn 0, so items with q=0 are excluded.Source: G Worked Samples · p. 26
Loading 2609.27395v1…