PaperlayerView exact revision on arXiv

Source-linked AI summary

Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language Aesthetic Judges

Amit Jadhav, Shaurya Beriwala, Beomjin Kim

arXiv:2609.27110v1cs.CVcs.CLcs.LG

TL;DR

Whether panels of VLM aesthetic judges outperform their best single member is supported by limited evidence. This paper compares holistic verdicts with fused scores on five frozen rubric dimensions, finding gains across EVA panels but parity on PARA.

  • Problem

    Reported gains from VLM judge panels are small, rarely tested for significance, and drawn largely from configurations that worked.

    Source: §1 · p. 1

  • Method

    The authors have each model score images holistically and on five frozen human-written rubric dimensions, then fuse scores across families with an out-of-fold combiner.

    Source: §1 · p. 1

  • Results

    Dimension fusion beat the best member in all 10 EVA three-family panels (pre-declared trio: +0.10 ρ), but reached ρ parity and was slightly below under τb on PARA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A cheap holistic probe of distance from the human noise ceiling can inform whether to deploy a panel before dimension scoring.

    Source: §1 · p. 2

    On EVA, gains require distinctiveness and emotional-impact rubric dimensions, which could not be validated against matching human attributes.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

Vision-language models (VLMs) are deployed as zero-shot judges of image aesthetics, and panels of several models are recommended, on thin evidence, as the way to make such judges reliable. On two human-rated datasets, EVA and PARA, we find that a panel of holistic judges never significantly beats its best member, whether the verdicts are averaged or fused by a learned combiner. What a panel is worth depends on what it is fed. We therefore have each model score each image on the five dimensions of a frozen, human-written rubric and fuse those scores, alongside each model's verdict, across model families with an out-of-fold combiner. The dimension scores measure what their labels claim: with the overall human score partialled out, a dimension prompt carries more attribute-specific information than the holistic prompt in 28 of 30 model-attribute cells. Fused, they beat the best single VLM in all ten three-family panels on EVA (against that best single model, +0.07 Spearman rho for the strongest trio and +0.10 for the pre-declared one, and +0.06 and +0.07 when averaged over twenty fold partitions; against the panel mean, the primary test gives +0.118 on its EVA design set), and on PARA they reach parity under Spearman rho and a small, non-significant loss under Kendall tau-b, where one model already captures 85% of the human noise ceiling. It is not a feature-count artefact: giving the same combiner an equal number of pure holistic columns, split from the same repetitions, does not reproduce it. The gain costs a few hundred labels, which do not transfer between datasets, and 4.8x the API calls on EVA; we report it with paired bootstraps and Kendall tau-b, alongside a failed pre-registration and the configurations that lost.

1 Introduction

The paper asks whether VLM aesthetic-judge panels improve when given rubric-dimension scores rather than holistic verdicts. Dimension fusion helps on EVA but reaches parity on PARA, while holistic verdict panels do not beat their best member.

1 Introduction

  • The proposed rubric-decomposed fusion scores each image on five frozen human-authored dimensions and fuses those scores with holistic verdicts across model families using an out-of-fold combiner.The dimensions are technical quality, composition, colour and tone, distinctiveness, and emotional impact.Source: §1 · p. 1
  • 10/10 EVA three-family panels beat their best member (pre-declared trio: +0.10 ρ); dimension fusion reaches ρ parity on PARA and is slightly below under τb.On PARA, one model already reaches 85% of the human noise ceiling.Source: §1 · p. 2
  • Holistic verdict fusion never beats the best single model on either dataset, even with more holistic columns, better averaging, or a learner suited to three features.The reported holistic-panel comparisons include both averaging and learned fusion.Source: §1 · p. 2
  • With overall human score partialled out, dimension prompts carry more attribute-specific information in 28/30 model–attribute cells, while raw correlations favor holistic prompts in 24/30.The contribution summary reports that 23 of the 30 differences survive correction.Source: §1 · p. 2

2 Related work

Prior work studies panels of holistic judges and decomposition within a single model backbone. This paper instead fuses scores from multiple model families using a frozen human-written rubric and validates them against human attribute ratings.

2 Related work

  • Prior judge panels average or combine verdicts, but reported gains are small and often compare against naive baselines rather than the panel’s best member.MAPLE reports +0.006 concordance over its best single evaluator without a paired test, and its analysis notes that individual models often beat the panel.Source: §2 · p. 2
  • Existing decomposition systems typically use one backbone and model-generated criteria; this work spans model families with a frozen human-written rubric and validates scores against human per-attribute ratings.Figure 1 shows the combiner receiving holistic scores, dimension scores, or both.Source: §2 · p. 2, Figure 1 · p. 3
  • The paper makes no claim that role framing improves judging and tests directly what the dimension prompts measure.The related-work discussion notes that role framing is an unreliable source of judging ability.Source: §2 · p. 3
  • Off-the-shelf image-aesthetic-assessment systems serve as baselines, but the EVA comparison is contaminated because 90% of its images lie in Q-Align’s AVA training split.Source: §2 · p. 3

3 Method

The method constructs holistic and rubric-dimension features for each image, then compares out-of-fold predictions from different feature blocks. A fixed combiner and shared comparators support the input ablation.

3.1 Problem formulation

  • Each image receives per-family holistic features and scores on five rubric dimensions; the combiner compares H, P, or H+P predictions out of fold using rank correlation.Dimension features average three calls per dimension; the holistic panel comparator is the unweighted mean of z-scored verdicts.Source: §3.1 · p. 3

3.2 Rubric decomposition

  • The five dimensions follow Leder et al.’s model of aesthetic appreciation, with five score tiers of observable criteria per dimension in an author-written rubric that was frozen before experiments.The rubric was never modified, and its SHA-256 hash was logged with every checkpoint.Source: §3.2 · p. 4

3.3 Combiner and comparators

  • A fixed gradient-boosting combiner uses 100 trees, depth 3, and learning rate 0.1; fused arms are compared against the best single model and the same-family panel mean.The combiner settings were fixed before fusion experiments and never tuned.Source: §3.3 · p. 4

4 Experimental setup

The experiments use two human-rated image datasets, five dimension-scored model families, and paired statistical comparisons. The EVA design-set and PARA confirmatory-set roles are distinguished in the protocol.

4 Experimental setup

  • EVA-500 and PARA-500 provide overall and per-attribute human ratings; five model families are dimension-scored, with three designated before the leaderboard and two added afterward.Both prompt types use temperature 0.3; dimension prompts use three repetitions, while holistic prompts use 25 on EVA and five on PARA.Source: §4 · p. 4
  • Performance is measured by Spearman ρ and Kendall τb against the human mean, with τb reported throughout because it is tie-aware.The noise ceiling is split-half reliability, corrected to the full rater count and square-rooted.Source: §4 · p. 4
  • Arm differences use paired bootstraps over images with 10,000 resamples, 95% percentile confidence intervals, one-sided p-values, and Benjamini–Hochberg correction for related comparisons.Claims selecting the best of several models also carry a max-statistic p-value.Source: §4 · p. 4
  • EVA is the design set for the primary fusion test, PARA is confirmatory, and results involving the two later-added families are labelled exploratory.Source: §4 · pp. 4–5
  • Mistral-Small-24B is the best single-model comparator on both datasets, scoring ρ = 0.598 on EVA and ρ = 0.818 on PARA.The EVA off-the-shelf baseline comparison is contaminated in Q-Align’s favour and measures transfer rather than held-out accuracy.Source: §4 · p. 5

5 Results

Dimension prompts carried more attribute-specific information than holistic prompts in most model-attribute cells, and cross-family fusion helped on EVA but not PARA. The results locate the gain in rubric dimensions, while limiting its scope through validation gaps, fold sensitivity, and dataset-specific training labels.

5.1 Dimension prompts measure their dimensions

  • Dimension prompts beat holistic prompts in partial correlation with the matched human attribute in 28 of 30 cells, but distinctiveness and emotional impact lack usable human ground truth.The partial correlation controls for the overall human score; 23 of 30 differences exclude zero after correction, while the two reversals have confidence intervals spanning zero.Source: §5.1 · p. 5

5.2 Fusion across families: the pre-declared test and an honest comparator

  • On EVA, primary H+PGB scored 0.693 vs 0.575 for the panel mean (∆ρ = +0.118 [+0.075, +0.163]); on PARA, it scored 0.820 vs 0.818 (+0.002).The comparison uses the pre-declared Gemini, Qwen2.5 and Mistral families; the strong trio’s fusion beat its best single model by +0.067 ρ [+0.023, +0.111].Source: §5.2 · pp. 5–6, Table 2 · p. 6

5.3 What the combiner is fed

  • On EVA, PGB beat HGB in every family block (+0.137, +0.080, +0.116) and the best single model in every block (+0.090, +0.053, +0.079).On PARA, PGB ranged from −0.012 to +0.003 against the best single model; adding verdicts to dimensions was worth a further +0.005 to +0.020, not significant after correction.Source: §5.3 · p. 6
  • Matching holistic panels to P’s feature count did not reproduce the EVA gain: block-split holistic features remained below the best single model, while P led by +0.07 to +0.15.Matching per-feature averaging preserved the gap; ridge improved HGB but still did not significantly beat the best single model on either dataset.Source: §5.3 · pp. 6–7

5.4 All ten panels

  • Across ten EVA trios, H+PGB beat the best member in all ten and survived correction in all ten; on PARA, only four trios survived correction, all excluding Mistral.The three best-fusing trios included both Mistral, the strongest holistic judge, and Gemini, the weakest, so holistic verdict strength did not identify the best fusion families.Source: §5.4 · p. 7

5.5 When fusion pays

  • The authors propose headroom as a candidate for when fusion pays: the best probe model captured 62.0% of EVA’s noise ceiling versus 84.6% on PARA.The predictor pattern rests on two datasets; agreement did not replicate across 20 trios, and headroom remains a hypothesis for a third dataset.Source: §5.5 · p. 7

5.6 Robustness, and what the labels buy

  • Across 20 fold partitions, EVA margins over the best single model averaged +0.071 ± 0.011, +0.059 ± 0.011, and +0.084 ± 0.011; PARA margins straddled zero.Labels did not transfer: EVA-trained fusion scored 0.747 on PARA against 0.818 for its best single model (−0.071 [−0.109, −0.037]); EVA precision@10% was 0.24 for the best single model versus 0.48–0.50 for H+PGB.Source: §5.6 · pp. 7–8

6 Ablations and negative results

The ablations show that feature selection and temperature assignment do not improve on the all-features fusion, while repeated scoring can improve Spearman correlation by breaking ties.

6 Ablations and negative results

  • Every tested feature-selection oracle fell short of the all-features combiner, while honest in-fold selection trailed it by +0.068 and +0.094.Every other tested configuration landed within ±0.015 of H+PGB.Source: §6 · p. 8
  • The pre-registered temperature hypothesis failed: its confirmatory inverted assignment gave −0.014 [−0.037, +0.008] on Gemini/PARA.The first temperature prediction held for only 1 of 3 dimensions; all dimension scoring therefore used a single temperature.Source: §6 · p. 8
  • Averaging 75 Gemini runs on EVA raised Spearman ρ from 0.440 to 0.468, while rounding the mean back to integers returned it to 0.437.The authors attribute the gain to breaking ties and also report results under Kendall τb.Source: §6 · p. 8

7 Discussion and limitations

The authors recommend probing headroom before deciding whether to pay for dimension scoring and fusion. The evidence is limited to two stratified 500-photo datasets, and the headroom boundary between them is unknown.

7 Discussion and limitations

  • A five-repetition holistic probe can guide deployment: use dimension-scored fusion when headroom is large, but the best single model when it is small.The EVA recipe costs 4.8× the calls and needs a few hundred target-distribution labels; the boundary between 62% and 85% headroom is unknown.Source: §7 · p. 8
  • The study covers two stratified datasets of 500 photographs, and whether the rubric transfers to other visual domains is untested.The combiner uses labels from the evaluated distribution; out-of-fold scoring bounds but does not eliminate this concern.Source: §7 · p. 8

8 Conclusion

The conclusion is that panel performance depends on its inputs: rubric dimension scores, unlike holistic verdicts alone, support gains on EVA but only parity on PARA.

8 Conclusion

  • Holistic verdict panels never significantly beat the best single model, while rubric-dimension fusion beats it across all ten EVA panels and reaches parity on PARA.The dimension scores are reported as attribute-specific where tested.Source: §8 · p. 8

A Configuration ladder and cost

The configuration ladder finds that the dimension-score input, rather than the combiner alone, accounts for the fusion gain; within-model decomposition also helps most when holistic scores are weaker.

A Configuration ladder and cost

  • Table 3 presents the EVA configuration ladder for n = 500, with all combiner arms evaluated by five-fold out-of-fold scoring and exploratory rows below the line.The listed ceiling is √Rfull = 0.947.Source: Table 3 · p. 11
  • The pre-declared all-features fusion remains ahead of exploratory alternatives: P+F11 is only +0.002 above it, and all five families are +0.004.The all-five-family difference is +0.004 [−0.018, +0.027].Source: A Configuration ladder and cost · p. 11
  • The dimension-block gain changes little with ridge, while ridge repairs HGB but not the verdict panel.Under ridge, PGB scores 0.670, 0.633, and 0.675 on EVA and 0.824, 0.832, and 0.825 on PARA.Source: A Configuration ladder and cost · p. 11
  • Figure 2 compares five inputs for one pre-declared trio and combiner, with paired 95% intervals against the best single model.Intervals clearing the dotted baseline indicate p < 0.05; feature counts appear in parentheses.Source: Figure 2 · p. 12
  • Feature selection never matched the all-features arm: even oracle selection fell short, while honest selection trailed by +0.068 and +0.094.The two reported gaps compare all-15 with per-dimension selection and all-18 with honest matrix selection, respectively.Source: A Configuration ladder and cost · p. 12
  • Figure 3 plots accuracy against API calls per image; on EVA, additional calls buy +0.10 ρ, while on PARA they buy nothing.Its points compare the best holistic model, that model with dimension scores, the pre-declared trio, and all five families.Source: Figure 3 · p. 12
  • Figure 4 sorts all ten three-family panels by fused ρ; EVA fusion beats the best member in all ten, while significant PARA gains occur only without Mistral.Fusion beats the panel mean by point estimate in all ten EVA trios; 10 and 8 of 10 EVA trios survive correction against the best member and panel mean, respectively.Source: Figure 4 · p. 13

B Decomposition within a single model

  • Within a single model, adding dimension means to the holistic mean yields smaller EVA gains as the holistic prompt improves; no PARA family gains under τb.On EVA, the reported gains range from +0.075 ρ for Gemini to +0.004 for Qwen3-VL-235B; Mistral loses −0.025 ρ on PARA.Source: B Decomposition within a single model · p. 13

C Single models against supervised baselines

Single holistic VLMs are competitive with established methods, but transfer comparisons are qualified by EVA’s overlap with AVA training data. Dimension prompts carry more attribute-specific information than holistic prompts in most tested cells, while holistic-pair combiner gains remain limited.

C Single models against supervised baselines

  • On PARA, four of seven single VLMs beat Q-Align after correction (+0.094 ρ); on EVA, the nominal +0.065 margin does not survive model selection.Under τb, Gemma-3-27B’s EVA margin is +0.093 because Q-Align’s discrete output ties many images.Source: C Single models against supervised… · p. 13
  • 451 of 500 EVA images appear in Q-Align’s AVA training list, so the baseline comparison measures transfer to EVA’s raters, not held-out accuracy.On the 49 held-out images, Q-Align scores 0.882; memorisation cannot be excluded at that sample size.Source: C Single models against supervised… · pp. 13–14

D Full multitrait–multimethod matrix

  • 28 of 30 matched dimension-prompt cells have higher partial correlations than holistic prompts, with 23 differences surviving Benjamini–Hochberg correction.The PARA quality partial is nearly degenerate: 96% of its attribute rank variance is shared with the overall score.Source: D Full multitrait–multimethod matrix · pp. 15–16, Table 5 · p. 16
  • The two partial-correlation reversals have confidence intervals spanning zero, so they are absence of evidence rather than evidence against dimension prompts.Both reversals occur on EVA Qwen3-VL colour and PARA Gemma-3-27B composition.Source: D Full multitrait–multimethod matrix · p. 16

E Diversity and gain across 28 pairs (EVA only)

  • Across 28 EVA model pairs, pair diversity’s regression coefficient for combiner gain is +0.020, but leave-one-model-out estimates span zero.No pair’s combiner beats its better member’s raw holistic score.Source: E Diversity and gain across 28 pairs… · p. 17

F Robustness and label-matched baselines

Across twenty fold partitions, the EVA fusion advantage over the best single model remains positive, while PARA comparisons cross zero. The fusion’s label budget does not transfer between datasets, and its shortlist advantage is strongest on EVA.

F Robustness and label-matched baselines

  • On EVA, H+PGB exceeds the best single model by +0.071 ± 0.011, +0.059 ± 0.011, and +0.084 ± 0.011; the comparison was negative in no partition.Source: F Robustness and label-matched baselines · p. 17
  • A same-label-budget CLIP embedding ridge probe matches H+PGB’s accuracy, but requires pixel access and produces no per-dimension scores.Training the combiner on EVA and applying it to PARA yields 0.747 versus PARA’s best single model at 0.818; the recipe cannot be amortised across datasets.Source: F Robustness and label-matched baselines · p. 17

G Top-decile utility

H Granularity and the choice of metric

The paper reports that fusion gains survive Kendall tau-b checks on EVA, while trio-level evidence does not establish whether agreement or headroom drives those gains. The temperature pre-registration mostly failed, and the rubric prompts use fixed, human-authored criteria.

H Granularity and the choice of metric

I The scope condition on twenty trios

  • Across 20 trios, agreement and headroom are too correlated to identify separately, leaving headroom a candidate rather than an established driver.The predictors correlate at r = +0.93, with max VIF 70.Source: I The scope condition on twenty trios · p. 18

J The temperature pre-registration

  • The temperature pre-registration’s P1 held for only one of three confident dimensions, P2 yielded −0.014 [−0.037, +0.008], and P3 did not apply.All dimension scoring in the paper uses temperature 0.3.Source: J The temperature pre-registration · p. 18

K Reproducibility

  • Hosted model endpoints are moving targets, so the paper logs model strings, temperatures, token counts, costs, and scoring dates in checkpoint records.The checkpoints contain 289,000 scored calls totalling $125.53.Source: K Reproducibility · pp. 18–19

L Prompts and rubric

  • Each dimension prompt supplies its rubric and asks for a score in the same JSON format, using criteria written before experiments and never modified.The five dimensions follow the stages of Leder et al.’s model of aesthetic appreciation.Source: L Prompts and rubric · p. 19
Loading 2609.27110v1…