PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?

Yichen Feng, Yuetai Li, Chunjiang Liu, Yuanyuan Chen, Fengqing Jiang, Yue Huang, Hang Hua, Zhengqing Yuan, Kaiyuan Zheng, Luyao Niu, Bhaskar Ramasubramanian, Basel Alomair, Xiangliang Zhang, Misha Sra, Zichen Chen, Radha Poovendran, Zhangchen Xu

arXiv:2605.12684v1cs.CVcs.AIcs.HC

TL;DR

Scalar image scores poorly capture comparative aesthetic preference, so the paper introduces VAB, an expert-grounded benchmark for selecting the best and worst images from matched-subject candidate sets. Across 20 frontier MLLMs and six reward models, the strongest system achieves 26.5% TB-1 pass^3 versus 68.9% for human experts, revealing a substantial performance gap.

  • Problem

    Existing aesthetic benchmarks typically infer preference from independent scalar image scores, limiting direct measurement of comparative execution quality.

    Source: §1 · p. 1

  • Method

    VAB evaluates comparative selection over matched-subject candidate sets using labels based on consensus from independent expert judges.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Models show a persistent gap from human experts, with the strongest system reaching 26.5% on TB-1 pass^3 versus 68.9% for humans.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    VAB provides a set-based, expert-grounded testbed for tracking the gap between multimodal models and expert aesthetic judgment.

    Source: §1 · p. 3, §6 · p. 10

    Human judgments were collected through voluntary online participation, and only aggregated benchmark results were released.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are now routinely deployed for visual understanding, generation, and curation. A substantial fraction of these applications require an explicit aesthetic judgment. Most existing solutions reduce this judgment to predicting a scalar score for a single image. We first ask whether such scores faithfully capture comparative preference: in a controlled study with eight expert annotators, score-derived rankings align poorly with the same annotators' direct comparisons, while direct ranking yields substantially higher inter-annotator agreement on best- and worst-image labels. Motivated by this finding, we introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter. VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task. Evaluating 20 frontier MLLMs and six dedicated visual-quality reward models, we find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts. Fine-tuning a 35B-parameter model on 2,000 expert examples brings its accuracy close to that of a 397B-parameter open-weight model, suggesting that the comparative signal in VAB is transferable. Together, these results expose a clear and measurable gap between current multimodal models and expert aesthetic judgment, and VAB provides the first set-based, expert-grounded testbed on which that gap can be tracked and closed.

1 Introduction

The paper argues that direct comparative ranking captures aesthetic preference more faithfully than scalar image scoring and introduces VAB as an expert-grounded, set-based benchmark. Across frontier models and reward models, VAB reveals a substantial, permutation-sensitive gap from human aesthetic judgment, while expert fine-tuning transfers effectively.

Motivation

  • Existing aesthetics benchmarks typically infer preference from independent scalar scores, mixing expertise, taste, and noise while allowing subject-content shortcuts.These limitations make it difficult to isolate execution quality across composition, color harmony, technical execution, and cultural convention.Source: §1 · p. 1

Human annotation study

  • Comparative ranking produced 42 percentage points higher best-image agreement than score-derived rankings, showing that direct comparison more faithfully captures aesthetic preference.Eight experts evaluated identical images using both independent scoring and direct ranking, and the protocols disagreed.Source: §1 · pp. 1–2

Benchmark design

  • VAB evaluates comparative selection over matched-subject candidate sets spanning 400 tasks, 1,195 images, 24 topics, and three visual domains.Each task contains two to six images, uses consensus from 10 expert judges with ambiguity filtering, and tests three random candidate-order permutations using ap@1 and pass^3.Source: §1 · pp. 2–3, Figure 1 · p. 2

Benchmark results

  • 26.5% of tasks were solved across all permutations by the strongest system, versus 68.9% for human experts, revealing a 42.4-point TB-1 pass^3 gap.Models retained only 34–65% of ap@1 accuracy across permutations and suffered a 7.1× drop with larger candidate sets, compared with 2.0× for humans.Source: §1 · pp. 2–3

Transferability

  • Fine-tuning a 35B-parameter model on 2,000 expert examples brought it close to a 397B-parameter open-weight model, indicating transferable comparative supervision.The benchmark evaluates 20 frontier MLLMs and six dedicated visual-quality reward models.Source: §1 · pp. 2–3

2 Human Annotation Study: Ranking vs. Scoring

A controlled study finds that absolute scoring poorly preserves annotators’ comparative preferences, whereas direct ranking produces more reproducible and distinguishable best-image labels. Accordingly, the authors adopt ranking for VAB annotation.

Which protocol yields a more scientifically sound signal for data annotation?

  • Eight experts evaluated 119 homogeneous-content and 107 heterogeneous-content tasks using independent scores si,a ∈[0, 10] or direct rankings of groups sized n ∈{2, 3, 4, 5}.The protocols were compared on fidelity, reproducibility, and distinguishability; reproducibility uses majority thresholds of at least 5 of 8 best-image votes.Source: Which protocol yields a more… · p. 3
  • Mean Kendall’s τ between scoring-derived rankings and direct comparisons is 0.188 for homogeneous tasks and 0.351 for heterogeneous tasks, with top-1 self-consistency of 45.5% and 52.0%.Nearly half of numeric score annotations fail to recover the order expressed by the same annotator under direct comparison.Source: Which protocol yields a more… · p. 3, Table 1 · p. 4
  • Ranking improves homogeneous-task reproducibility by 42.0 points for accbest, 37.0 points for accworst, and 51.3 points for accbest&worst, all with p < 0.001.The same pattern holds for heterogeneous-content tasks, indicating that independent experts reproduce best/worst labels more reliably under ranking.Source: Which protocol yields a more… · pp. 3–4, Table 2 · p. 4
  • Ranking concentrates top-choice votes more strongly, lowering entropy from 1.213 to 0.729 and increasing unanimous choices from 0.8% to 10.1%.The difference is statistically significant with Wilcoxon p = 1.7 × 10−15.Source: Which protocol yields a more… · p. 4
  • The authors attribute ranking’s advantage to relational aesthetic judgment and the lower cognitive burden of comparative evaluation than absolute scoring.Absolute scoring requires an internal reference scale and conversion of qualitative impressions into numbers, increasing instability.Source: Which protocol yields a more… · p. 4

3 The VAB Benchmark

VAB evaluates expert-aligned aesthetic judgment through comparative selection among matched-subject candidate sets rather than scalar image ratings. It covers multiple visual domains and uses repeated randomized evaluations with direct expert-derived labels.

3.1 Overview

  • VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, spanning 24 topics with candidate sets of 2–6 images.Top-1 selects the best image, while TB-1 selects both the best and worst.Source: §3.1 · p. 5
  • Labels come from direct comparative judgments by independent experts, with each task evaluated under three random candidate-order permutations using ap@1 and pass^3.pass^3 counts a task as correct only when all three evaluations are correct.Source: §3.1 · p. 5

3.2 Data Construction

  • VAB constructs matched image groups whose members share subject matter but vary in execution quality, organizing data collection around comparative ranking.The benchmark contrasts with prior evaluations based on aggregated absolute ratings.Source: §3 · p. 4, §3.2 · p. 5
  • The data pipeline includes commissioned fine-art sets and photography sets expanded through expert-retouching and scalable agent-editing pipelines.Fine art includes 426 sets and 1,142 images, while photography includes 670 sets and 1,809 images.Source: §3.2 · p. 5

Step 1: Data Collection by Domain

VAB constructs matched-subject candidate sets across illustration domains and evaluates them through blind, comparative expert annotation. Tasks are retained only when expert votes for the best and, when applicable, worst image meet statistically defined concentration thresholds.

Step 1: Data Collection by Domain

  • VAB includes 271 illustration sets comprising 908 images, generated from text-to-image prompts or controlled renderings of CC0 3D assets with matched subject matter and varied execution.All collected sets are deduplicated and checked for semantic consistency before annotation.Source: Step 1: Data Collection by Domain · p. 6

3.3 Expert Annotation and Ground Truth

  • The review uses 10 independent expert judges per candidate set, drawn from more than 100 evaluators, with judges blind to artist identity and over 13,000 assessments completed.Creation and evaluation are strictly separated, and the review involved more than 300 hours of annotation.Source: §3.3 · p. 6
  • Each task presents k candidate images side by side; judges assess them with domain-specific rubrics and select the best image for k = 2, or both best and worst images for k ≥3.Every task receives exactly 10 judgments, with distinct best and worst choices required when k ≥3.Source: §3.3 · p. 6
  • Tasks are retained only when expert votes meet concentration thresholds: best-image votes for k = 2, and both best- and worst-image votes for k ≥3, under a uniform-random statistical null.Satisfying only one threshold is insufficient when k ≥3.Source: §3.3 · p. 7

4 Experiments

Experiments evaluate frontier MLLMs and reward models on comparative aesthetic selection against human and random baselines. Models remain substantially below experts, with performance further limited by domain difficulty, instability, and larger candidate sets.

4.1 Setup

  • The evaluation covers 20 MLLMs and six reward models, using three random candidate-order permutations and comparing systems with fresh 10-expert and analytical random-guess baselines.MLLMs directly select the best or both best and worst images; reward models score images independently before within-task ranking.Source: §4.1 · p. 7

4.2.1 Results for Multimodal Large Language Models

  • 26.5% TB-1 pass^3 for Claude Sonnet 4.6 trails human experts’ 68.9%, while Doubao Seed 2.0 Pro reaches 42.8% TB-1 ap@1 versus 68.9% for experts.These are the main VAB results for 20 frontier MLLMs, reported in Table 4.Source: §4.2.1 · p. 8, Table 4 · p. 8
  • Model progress is uneven: Claude Sonnet 4.6 nearly doubles its predecessor from 14.5% to 26.5% TB-1 pass^3, while GPT-5.2 falls to 15.5% and Qwen3.5-397B-A17B reaches 17.2%.Under the stricter metric, the strongest open-weight system remains below Claude Sonnet 4.6’s 26.5%.Source: §4.2.1 · p. 8
  • Illustration is hardest, with Claude Sonnet 4.6 at 19.0% TB-1 pass^3, while photography reaches 30.2% for o4-mini; GLM-4.6V drops from 34.1% to 11.5% across metrics.The metric gap indicates sensitivity to candidate order and response instability.Source: §4.2.1 · pp. 8–9

4.2.2 Results for Reward Models

  • Q-Align is the strongest overall reward model, reaching 52.2% Top-1 and 38.2% TB-1 accuracy, versus human baselines of 77.7% and 68.9%, respectively.Source: §4.2.2 · p. 9

4.3 Ablations and Additional Analysis

  • Under TB-1 pass^3, the best model declines from 47.3% on two-image tasks to 6.7% on four-image tasks, while human experts fall from 87.1% to 43.6%.Larger candidate sets amplify comparative difficulty and instability.Source: §4.3 · p. 9

5 Fine-Tuning on Expert Aesthetic Judgments

LoRA fine-tuning QWEN3.5-35B-A3B on 2,000 disjoint expert-annotated examples improves all reported VAB metrics. The resulting KALLISTI-35B-A3B nearly matches the much larger Qwen3.5-397B-A17B, indicating transferable comparative supervision.

Training Setup

  • KALLISTI-35B-A3B uses LoRA with rank 64 and α = 128, freezing the vision tower and updating the multimodal projector on 2,000 disjoint expert examples.Training lasts three epochs with a learning rate of 1 × 10−4 and cosine scheduling.Source: §5 · p. 9

Improvement over the base model

  • 29.5% versus 25.5% Top-1 pass^3, 17.5% versus 15.8% TB-1 pass^3, 51.3% versus 47.5% Top-1 ap@1, and 38.8% versus 36.5% TB-1 ap@1 after fine-tuning.Table 6 reports consistent gains over the base model across all four metrics.Source: Table 6 · p. 10, §5 · p. 10

Scaling implication

  • 29.5% versus 29.8% on Top-1 pass^3 places KALLISTI-35B-A3B near Qwen3.5-397B-A17B, within roughly one point on the other metrics.The result suggests that VAB’s comparative supervision can compensate for an order-of-magnitude parameter reduction under lightweight adaptation.Source: §5 · p. 10

6 Conclusion

VAB reframes aesthetic evaluation as expert-grounded comparative selection over matched candidate sets, exposing a substantial gap between frontier multimodal models and human judgment. Its results also show that comparative supervision transfers efficiently through lightweight fine-tuning, while motivating broader domains, stronger permutation robustness, and larger expert datasets.

Benchmark results

  • 26.5% of tasks passed TB-1 pass^3 for the strongest system, versus 68.9% for human experts, across 20 frontier MLLMs and six reward models.Accuracy also drops sharply as candidate sets grow, and models show pronounced positional instability across candidate-order permutations.Source: §6 · p. 10
  • Fine-tuning a 35B-parameter model on 2,000 expert examples brought its accuracy close to that of a 397B-parameter open-weight model.The result suggests that expert comparative supervision transfers efficiently with lightweight adaptation.Source: §6 · p. 10

Benchmark design

  • VAB recasts aesthetic evaluation as comparative selection over matched-subject candidate sets, using expert-ranking labels across 400 tasks and 1,195 images.The benchmark spans fine art, photography, and illustration.Source: §6 · p. 10

Human study

  • Direct ranking produced 42 percentage points higher inter-annotator agreement than score-derived rankings in the controlled human study motivating VAB.This supports replacing scalar scoring with comparative expert ranking for the benchmark protocol.Source: §6 · p. 10

Future work

  • Future work should broaden domain and annotator diversity, improve robustness to candidate-order permutations, and scale expert comparative supervision to larger training corpora.VAB establishes expert aesthetic judgment as a measurable comparative target for systematically pursuing expert-aligned multimodal evaluation.Source: §6 · p. 10

Ethics Statement … B.2 Raw Collection Statistics Before Expert Filtering

The paper frames VAB as an expert-grounded, set-based alternative to pointwise aesthetic scoring, while documenting ethical safeguards, benchmark construction, and supporting appendix materials. The benchmark spans multiple visual domains and reports both finalized and pre-filtering collection statistics.

Ethics Statement

  • The study received an exempt human-subjects determination for minimal-risk behavioral research, with voluntary participation, no sensitive-data collection, and only aggregated public results.The University of Washington Human Subjects Division reviewed the protocol under Categories 2 and 3 of U.S. federal regulations.Source: Ethics Statement · p. 10
  • Benchmark materials were commissioned, licensed, procedurally generated, or built from CC0 assets, with commitments to creator credit and fair compensation.Agent-edited photographs remained within source-licensing agreements, while illustration prompts avoided reproducing copyrighted material.Source: Ethics Statement · p. 11

Appendix

  • The appendix provides topic-level annotation examples organized into fine art, photography, and illustration sections.These materials appear in Appendix I and its domain-specific subsections.Source: Appendix · p. 17

A Related Work

  • Prior aesthetic resources mainly predict scalar scores for isolated images, inferring preference order only indirectly from independent estimates.Existing datasets and models include AVA, AADB, NIMA, Q-Align, and UniQA.Source: A Related Work · p. 17
  • VAB targets relative aesthetic discrimination by selecting strongest and weakest images from matched-subject candidate sets across fine art, photography, and illustration.Labels derive from expert consensus rather than crowd averages, distinguishing the benchmark from predominantly single-image, score-centric resources.Source: A Related Work · p. 17

B.1 Final Benchmark Statistics

  • The final benchmark statistics are reported after expert annotation and consensus filtering, including domain, topic, and candidate-set-size breakdowns.Tables 7–9 cover finalized benchmark statistics, candidate-set sizes, and topic-level distributions.Source: §B.1 · p. 17, Table 7 · p. 18, Table 8 · p. 18

B.2 Raw Collection Statistics Before Expert Filtering

  • Before expert filtering, the collection contained 1,367 tasks and 3,859 images, with raw summaries by domain, topic, and photography pipeline.Tables 10–12 provide the corresponding decompositions.Source: §B.2 · p. 17

B.3 Data Construction Details … D.1 Selected Thresholds

The benchmark constructs matched-subject candidate sets across fine art, photography, and illustration, then uses domain-specific comparative rubrics, side-by-side expert selection, and consensus filtering to establish ground truth. The resulting protocol evaluates aesthetic quality through best- and, for larger sets, worst-image judgments rather than separate numerical scores.

B.3 Data Construction Details

  • The dataset spans commissioned fine-art sets, controlled photography edits, and generated or rendered illustration sets, with deduplication and semantic-consistency review before annotation.Fine art includes 426 sets and 1,142 images; photography includes 670 sets and 1,809 images; illustration includes 271 sets and 908 images.Source: §B.3 · pp. 18–19

C.1 Annotation Rubrics

  • The annotation rubrics guide structured image assessment but do not function as separate numerical scoring forms; judges apply domain criteria before making a final comparative selection.This protocol treats the rubric as a comparison guide rather than a scalar rating instrument.Source: §C.1 · p. 21

Fine Art Rubric

  • Fine-art rubrics compare paintings with matched style, subject, and medium using composition, color, spatial organization, brushwork, finish, atmosphere, and overall artistic persuasiveness.Judges also identify the strongest work and, for sets of at least three, the weakest work while ignoring personal subject preference and external context.Source: Fine Art Rubric · p. 21

Photography Rubric

  • Photography rubrics compare closely matched scenes through subject emphasis, framing, background, lighting, tonal and color control, focus, technical accuracy, and emotional or narrative strength.The rubric separately asks for the strongest image and, when applicable, the weakest, while excluding judgments based on subject attractiveness, shooting difficulty, or post-processing effort.Source: Photography Rubric · p. 22

Digital Illustration Rubric

  • Digital-illustration rubrics assess visual hierarchy, flow, structural accuracy, pose refinement, palette and lighting coherence, detail control, stylistic alignment, consistency, and use-case suitability.Judges select the strongest illustration and, for sets of at least three, the weakest, while excluding personal style preference, software choice, and hypothetical client requirements.Source: Digital Illustration Rubric · pp. 22–23

C.2 Annotation Interface

  • The interface presents all k candidates side by side, with judges selecting the best image for k = 2 and both the best and worst images for k ≥3.Judges first review each candidate against the rubric criteria, then submit the comparative choice.Source: Figure 5 · p. 24, §C.2 · p. 24

D Ground Truth Mathematical Details

  • The consensus filter uses mathematically defined thresholds to determine accepted ground-truth labels from expert judgments.The appendix supplies additional details for the consensus filter, while Table 13 reports thresholds and corresponding null pass probabilities for candidate-set size k with n = 10 judges per task.Source: D Ground Truth Mathematical Details · p. 24, §D.1 · p. 24, Table 13 · p. 24

D.1 Selected Thresholds

  • Selected thresholds are reported together with their corresponding null pass probabilities for each candidate-set size, using 10 judges per task.These thresholds define the benchmark’s expert-consensus filtering conditions.Source: §D.1 · p. 24, Table 13 · p. 24

D.2 Derivation of Null Pass Probabilities … E.2 Comprehensive Reward-Model Results

The appendix derives and validates null pass probabilities, handles binary tasks separately, and provides comprehensive MLLM and reward-model results across overall and domain-level VAB evaluations. These results expose order sensitivity, domain variation, and asymmetric strengths in identifying best versus worst images.

D.2 Derivation of Null Pass Probabilities

  • For k ≥3, null pass probabilities enumerate directed best-worst vote configurations and require both best- and worst-vote thresholds to be met.The exact probability sums multinomial mass over valid configurations satisfying the conjunctive acceptance condition.Source: §D.2 · pp. 24–25

D.3 Monte Carlo Verification

  • Monte Carlo verification uses 107 samples per (k, m) configuration and compares estimates with exact probabilities using 95% Wald confidence intervals.Table 14 presents the verification against the exact values from Table 13.Source: §D.3 · p. 25, Table 14 · p. 25

D.4 Special Case for Binary Tasks

  • For binary tasks, the worst image is implied by the best, so the filter requires maxi Bi ≥m; with n = 10 and m = 8, pass probability is approximately 10.94%.The lower binary threshold avoids discarding disproportionately many informative comparisons because each annotator makes only one comparative decision.Source: §D.4 · p. 25

E Detailed Experimental Results

  • The detailed experimental appendix expands evaluation to overall performance, domain variation, candidate-order sensitivity, decoding, and task-size analyses.It retains the main paper’s evaluation focus while supplying full tables too large for the main text.Source: E Detailed Experimental Results · p. 25

E.1 Comprehensive MLLM Results

  • Comprehensive MLLM results show pervasive gaps between ap@1 and pass^3, indicating candidate-order sensitivity and response instability across models.Table 15 reports Top-1, Bot-1, and TB-1 under both metrics across overall and domain-level evaluations.Source: §E.1 · p. 25, Table 15 · p. 26
  • The MLLM results show Claude Sonnet 4.6 improving over 4.5 on TB-1 pass^3, while GPT-5 through GPT-5.2 declines monotonically.Qwen3.5-397B-A17B is closer to the strongest closed models under ap@1 than pass^3, suggesting weaker repeated-evaluation consistency.Source: §E.1 · pp. 25–26
  • Across domains, illustration is most difficult, photography generally most tractable, and fine art intermediate for the evaluated MLLMs.The domain columns reinforce the main-text headline about domain-level difficulty.Source: §E.1 · p. 26

E.2 Comprehensive Reward-Model Results

  • Comprehensive reward-model results reveal that some models reject poor images more effectively than they identify the best, especially Q-Instruct in photography.The detailed Bot-1 columns make this asymmetry explicit relative to the condensed main-text results.Source: §E.2 · p. 26

E.3 Topic-Level MLLM Results … F Human Study Details

Supplementary analyses show that aesthetic-selection performance varies substantially by topic and candidate-set size, while decoding temperature affects stability without changing the benchmark’s central limitation. MLLMs and reward models perform best on selected photography topics, but illustration remains broadly difficult, especially under stricter consistency metrics.

E.3 Topic-Level MLLM Results

  • Across Top-1 ap@1, Top-1 pass^3, TB-1 ap@1, and TB-1 pass^3, illustration is the hardest MLLM domain, while photography is more tractable but uneven across topics.These topic-level trends are not driven by a single topic or metric.Source: §E.3 · p. 26
  • MLLM topic effects concentrate strength in Chinese painting, portrait-color, anime-and-manga, comic, macro, and sports tasks, while landscape-color, sketch-based, concept-art, digital-and-AI-art, stylized-3D, night-and-astrophotography, architecture, and some street-and-city tasks remain unstable.The harder cases are especially unstable under stricter criteria or repeated-evaluation consistency requirements.Source: §E.3 · p. 26
  • Topic-level results are reported in separate tables for fine art, illustration, and photography across Top-1 ap@1, Top-1 pass^3, TB-1 ap@1, TB-1 pass^3, Top-1, and TB-1 metrics.Source: §E.3 · p. 27, §E.4 · p. 28

E.4 Topic-Level Reward-Model Results

  • Reward-model gains are concentrated in selected photography topics, whereas illustration remains difficult across most models and topics, with many cases near random under TB-1.Fine-art performance is mixed rather than uniformly weak, with competitive results on Chinese painting and portrait-color tasks; photography gains do not transfer uniformly to best-and-worst selection.Source: §E.4 · p. 28

E.5 Temperature Ablation

  • Greedy decoding modestly improves stability overall, but its effect varies substantially across models and does not change the conclusion that comparative judgment, rather than decoding noise alone, is the dominant limitation.Gemini 3.1 Pro and Claude Sonnet 4.5 benefit most, especially on TB-1 pass^3, while Claude Sonnet 4.6, Doubao Seed 2.0 Pro, GLM-4.6V, and Qwen3.5-397B-A17B are flat or slightly worse on at least one metric.Source: §E.5 · pp. 28–29

E.6 Results by Candidate Set Size

  • Performance drops sharply as candidate-set size k increases, especially under TB-1 pass^3, widening the gap between models and human experts.Table 35 reports pass^3 and ap@1 across candidate-set sizes for evaluated MLLMs and reference baselines.Source: Table 35 · p. 37, §E.6 · p. 38

F.1 Detailed Setup and Metric Definitions … G.1 Hyperparameters

This section details the study setup, domain-specific ranking rubrics, evaluation metrics, statistical tests, and task construction for homogeneous and heterogeneous image comparisons. It also specifies the hyperparameters used to fine-tune KALLISTI-35B-A3B.

F.3 Task Examples

  • Tasks include 119 homogeneous-content groups and 107 heterogeneous-content regroupings, with candidates either sharing source content or being regrouped across topic-consistent source groups.Figure 6 contrasts independent single-image scoring with joint ordering of the full candidate set in these two settings.Source: §F.1 · p. 38, Figure 6 · p. 39

F.2 Extended Interpretation

  • The interpretation frames ranking as a more direct and cognitively natural comparative judgment, whereas scoring adds calibration, scale-use, and measurement noise.Comparative judgment can be made before specifying an absolute magnitude, while scoring requires an internal reference scale and numeric conversion.Source: §F.2 · p. 39

F.4 Annotation Protocol

  • Under scoring, annotators independently rated all 400 images without seeing group members, while ranking presented every group simultaneously for complete best-to-worst ordering.Score-derived rankings were later computed for both homogeneous and heterogeneous tasks; no exact within-task score ties were observed.Source: §F.4 · p. 40, Digital Illustration Single-Image… · p. 42

Fine Art Single-Image Scoring Rubric

  • The fine-art scoring rubric evaluates artistic completeness and maturity through subject clarity, composition, spatial arrangement, color, mark-making, technique, resolution, atmosphere, character, and persuasiveness.Scores are based solely on the visible image and exclude personal subject preference and contextual information about the artist or intent.Source: Fine Art Single-Image Scoring Rubric · p. 40

Photography Single-Image Scoring Rubric

  • The photography rubric scores photographic language and visual expression using subject emphasis, composition, lighting, exposure, focus, color, technical quality, moment, narrative, and presence.It evaluates the image itself while excluding the subject’s personal appeal and assumptions about shooting difficulty or post-processing effort.Source: Photography Single-Image Scoring Rubric · p. 41

Digital Illustration Single-Image Scoring Rubric

  • The digital-illustration rubric assesses visual design and professional standards through focal clarity, flow, form, palette, lighting, detail, edges, stylistic coherence, consistency, and use-case suitability.It excludes personal style preference, software or tools used, and hypothetical client requirements.Source: Digital Illustration Single-Image… · pp. 41–42

Fine Art Ranking Rubric

  • Fine-art ranking compares paintings sharing style, subject matter, and medium by relative differences in clarity, composition, color, technique, resolution, atmosphere, overall presence, and persuasiveness.The comparative rubric is designed for images differing subtly in execution and ranks them from best to worst.Source: Fine Art Ranking Rubric · pp. 42–43
  • Fine-art evaluation orders the complete image set from strongest to weakest while excluding personal subject preference and contextual information beyond the image.Source: Fine Art Ranking Rubric · p. 43

Photography Ranking Rubric

  • Photography ranking compares matched-subject images using composition, lighting, tonal control, focus, color, technical quality, and emotional or narrative impact.Annotators rank photographs from best to worst while excluding personal subject appeal and assumptions about shooting or post-processing effort.Source: Photography Ranking Rubric · pp. 43–44

Digital Illustration Ranking Rubric

  • Digital illustration ranking evaluates visual design and professional quality through focal clarity, visual flow, structural accuracy, coherence, technical control, style alignment, and use-case suitability.The rubric excludes personal style preference, software or tools used, and hypothetical client requirements.Source: Digital Illustration Ranking Rubric · p. 44

F.5 Notation

  • Each task contains n ∈{2, 3, 4, 5} images, with absolute scores, score-derived and direct rankings, best/worst labels, and accbest&worst measuring simultaneous satisfaction of both majority conditions.Source: §F.5 · p. 45

F.6 Statistical Tests

  • All six Bonferroni-corrected protocol-comparison tests remain significant at p < 0.001, with p-values as low as 1.0 × 10−14.Exact two-sided McNemar tests cover best, worst, and joint majority agreement in homogeneous and heterogeneous tasks; the Wilcoxon test for entropy yields p = 1.7 × 10−15.Source: §F.6 · p. 45

Homogeneous-Content Tasks (119 Tasks)

Heterogeneous-Content Tasks (107 Tasks)

  • Heterogeneous-content tasks retain 107 groups after excluding 12 of 119 regrouped tasks whose score-based Kendall’s W is at least 0.825.Each retained task is formed by regrouping images from distinct homogeneous-content tasks, typically within the same topic.Source: Heterogeneous-Content Tasks (107 Tasks) · p. 48

G.1 Hyperparameters

  • Fine-tuning KALLISTI-35B-A3B uses the hyperparameters specified in Table 39.The supplied material identifies Table 39 as the hyperparameter specification for this fine-tuning procedure.Source: §G.1 · p. 51, Table 39 · p. 52

G.2 Detailed Results

The detailed results evaluate fine-tuning across four metrics and show that its benefits vary by topic and domain. KALLISTI-35B-A3B improves most on several fine-art topics, remains competitive on illustration, and does not consistently outperform its base model in photography.

Topic-level results

  • Fine-tuning gains are uneven across topics: KALLISTI-35B-A3B improves most clearly on several fine-art topics, remains competitive with larger open-weight baselines on illustration, and does not uniformly dominate its base model in photography.The breakdown covers fine art, illustration, and photography under Top-1 ap@1, Top-1 pass^3, TB-1 ap@1, and TB-1 pass^3.Source: §G.2 · p. 52
  • The topic-level breakdown is organized by domain for Top-1 ap@1, Top-1 pass^3, TB-1 ap@1, and TB-1 pass^3, with separate tables for fine art, illustration, and photography.Tables 40–51 report the corresponding domain-specific topic results and domain-level averages.Source: §G.2 · p. 52

H Prompt Templates and Scoring Interfaces … Stylized 3D

The appendix specifies prompt templates and scoring interfaces for photography, illustration, MLLM evaluation, and reward-model evaluation. It also documents representative annotation examples covering VAB’s 24 topics.

H Prompt Templates and Scoring Interfaces; I Topic-Level Annotation Examples

H.1 Photography Pipeline Prompts

  • The photography pipeline uses eight independent ArtiMuse critiques, GPT-based edit planning through LiteLLM, and Gemini-3-Pro-Image synthesis of better and worse variants.The eight aspect-wise responses are concatenated into the review passed to planning; Round 2 conditions worse-image editing on the better image and uses the runtime file-backed template.Source: §H.1 · p. 56, §H.1.2 · p. 56, §H.1.3 · p. 58

H.2 Illustration Pipeline Prompts

H.3 MLLM Evaluation Prompts

  • MLLM evaluation prompts ask models to select the best image, the worst image, or both within each candidate set.These prompts are generic and vary with the task setting.Source: §H.3 · p. 59

H.4 Reward-Model Scoring Interfaces

  • Reward models score images independently using explicit, implicit, or no prompts; averaged repeated scores are converted into within-task rankings for predicted best and worst images.Each image is scored num_trials times before averaging.Source: §H.4 · p. 60

H.4.1 ArtiMuse and H.4.2 Q-Align

  • ArtiMuse and Q-Align use implicit prompts, with ArtiMuse decoding a two-letter output to [0, 100] and Q-Align weighting softmax probabilities over five quality words.Q-Align uses its aesthetic scorer by default, with weights [1, 0.75, 0.5, 0.25, 0].Source: §H.4.1 · pp. 60–61, §H.4.2 · p. 61

H.4.3 Q-Instruct and H.4.4 RealQA

  • Q-Instruct uses an explicit image-token-wrapped prompt and scores images by the logit difference between good and poor, whereas RealQA parses generated text into scores in [0.0, 10.0].RealQA issues a second constrained prompt when parsing fails.Source: §H.4.3 · p. 61, §H.4.4 · p. 61

H.4.5 PEAS

  • PEAS-aes and PEAS-ava are prompt-free vision models whose 10-bin score distributions are reduced to expected values.They provide pure vision scoring without language instructions.Source: §H.4.5 · p. 61

Related papers

Loading 2605.12684v1…