PaperlayerView exact revision on arXiv

Source-linked AI summary

VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations

Yupeng Xie, Zhiyang Zhang, Yifan Wu, Sirong Lu, Jiayi Zhang, Zhaoyang Yu, Jinlin Wang, Sirui Hong, Bang Liu, Chenglin Wu, Yuyu Luo

arXiv:2510.22373v3cs.CLcs.AIcs.CV

TL;DR

Visualization quality requires judging data fidelity, communication, and aesthetics, but existing benchmarks do not systematically evaluate MLLMs across these aspects. VisJudge-Bench provides such a benchmark, and VisJudge improves alignment with human experts compared with GPT-5.

  • Problem

    Existing benchmarks do not systematically assess MLLMs’ ability to evaluate visualization quality across data accuracy, information communication, and visual design.

    Source: §1 · p. 2

  • Method

    The authors build VisJudge-Bench with 3,090 expert-scored samples across visualization types and fine-tune VisJudge for visualization quality assessment.

    Source: §1 · p. 2

  • Results

    VisJudge achieved a 23.9% reduction in MAE to 0.421 and a 60.5% improvement in correlation to 0.687 compared with GPT-5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VisJudge-Bench provides a standardized evaluation platform, and VisJudge’s results support domain-specific training as a viable approach to improving MLLM evaluation capabilities.

    Source: §6 · p. 10

    The benchmark focuses on static visualizations, often lacks raw data for fidelity verification, and uses point estimates that may not capture diverse human preferences.

    Source: §7 · p. 10

Abstract

from arXiv · show

Visualization, a domain-specific yet widely used form of imagery, is an effective way to turn complex datasets into intuitive insights, and its value depends on whether data are faithfully represented, clearly communicated, and aesthetically designed. However, evaluating visualization quality is challenging: unlike natural images, it requires simultaneous judgment across data encoding accuracy, information expressiveness, and visual aesthetics. Although multimodal large language models (MLLMs) have shown promising performance in aesthetic assessment of natural images, no systematic benchmark exists for measuring their capabilities in evaluating visualizations. To address this, we propose VisJudge-Bench, the first comprehensive benchmark for evaluating MLLMs' performance in assessing visualization aesthetics and quality. It contains 3,090 expert-annotated samples from real-world scenarios, covering single visualizations, multiple visualizations, and dashboards across 32 chart types. Systematic testing on this benchmark reveals that even the most advanced MLLMs (such as GPT-5) still exhibit significant gaps compared to human experts in judgment, with a Mean Absolute Error (MAE) of 0.553 and a correlation with human ratings of only 0.428. To address this issue, we propose VisJudge, a model specifically designed for visualization aesthetics and quality assessment. Experimental results demonstrate that VisJudge significantly narrows the gap with human judgment, reducing the MAE to 0.421 (a 23.9% reduction) and increasing the consistency with human experts to 0.687 (a 60.5% improvement) compared to GPT-5. The benchmark is available at https://github.com/HKUSTDial/VisJudgeBench.

1 INTRODUCTION

VisJudge-Bench targets the gap in systematic evaluation of MLLMs’ ability to judge visualization quality across fidelity, expressiveness, and aesthetics. The paper also introduces VisJudge, which improves agreement with human experts over GPT-5.

1 INTRODUCTION

  • Visualization quality depends on faithful data presentation, clear communication, and aesthetic design, three interconnected dimensions that make assessment challenging.Source: §1 · p. 1
  • Existing benchmarks leave a gap: chart-QA assesses chart understanding, image-aesthetics benchmarks overlook data communication, and visualization benchmarks largely focus on NL2VIS fidelity.Source: §1 · p. 2
  • On VisJudge-Bench, even advanced MLLMs show substantial gaps from human experts, with MAE as high as 0.553 and correlation of only 0.428.The benchmark contains 3,090 expert-scored real-world samples across single visualizations, multiple visualizations, and dashboards, spanning 32 chart types.Source: §1 · p. 2
  • 23.9% lower MAE and 60.5% higher correlation with human experts: VisJudge reaches 0.421 MAE and 0.687 correlation compared with GPT-5.VisJudge performed best among the tested models.Source: §1 · p. 2

2 RELATED WORK

Prior work assesses visualization quality through recommendation, NL2VIS generation, or MLLM judging, but each addresses only part of comprehensive visualization design quality.

2 RELATED WORK

  • Rule-based and learning-based visualization recommendation methods rank charts, but rely on inflexible heuristics or limited evaluation dimensions and costly annotated data.Source: §2 · p. 3
  • NL2VIS benchmarks assess whether systems translate natural-language queries into visualizations, emphasizing code generation rather than judging visualization quality.Source: §2 · p. 3
  • MLLM-as-a-Judge work spans natural-image aesthetics, chart understanding, and visualization evaluation, yet these categories respectively overlook data communication, overall design, or intrinsic design quality.Source: §2 · p. 3

3 VISJUDGE-BENCH: DESIGN AND CONSTRUCTION

VisJudge-Bench is built through data collection and processing, adaptive evaluation-question generation, and expert annotation with quality control.

3 VISJUDGE-BENCH: DESIGN AND CONSTRUCTION

  • The benchmark construction follows three stages: data collection and processing, adaptive question generation, and expert annotation with quality control.Source: §3 · p. 3

3.1.1 DATA COLLECTION AND PREPROCESSING

  • From over 300,000 initial images, filtering and human verification yielded 13,220 valid visualizations; stratified sampling selected 3,090 across three categories and 32 subtypes.The final sample includes 1,041 single visualizations, 1,024 multiple visualizations, and 1,025 dashboards.Source: §3.1.1 · p. 4

3.1.2 THE “FIDELITY, EXPRESSIVENESS, AND AESTHETICS” EVALUATION FRAMEWORK

  • The framework operationalizes fidelity, expressiveness, and aesthetics as six dimensions: data fidelity, semantic readability, insight discovery, design style, visual composition, and color harmony.Because raw data are typically unavailable for web-collected visualizations, data fidelity is assessed from visually detectable presentation accuracy.Source: §3.1.2 · p. 5
  • GPT-4o extracts chart metadata and uses it to populate templates with tailored questions and five-point scoring criteria for the six evaluation sub-dimensions.Questions and criteria target visualization-specific features, such as baseline choice and proportional bar lengths when assessing data fidelity.Source: §3.1.2 · p. 5

3.1.3 EXPERT ANNOTATION AND QUALITY CONTROL

  • Three independent annotators scored each sample across six dimensions; disagreement screening generated candidate resolutions for expert review, modification, rejection, or consensus.Three visualization experts reviewed all samples, producing the human scoring benchmark used as the gold standard.Source: §3.1.3 · p. 6

4 VISJUDGE: A SPECIALIZED MODEL FOR VISUALIZATION EVALUATION

The authors develop VisJudge by fine-tuning open-source multimodal models on VisJudge-Bench, then evaluate them alongside baseline models on a shared test set.

4 VISJUDGE: A SPECIALIZED MODEL FOR VISUALIZATION EVALUATION

  • VisJudge investigates whether domain-specific fine-tuning improves visualization evaluation, comparing fine-tuned open-source models with closed- and open-source baselines on the same test set.Source: §4 · p. 6
  • Training uses a stratified 70%/10%/20% split of 2,163/279/648 samples, keeping training data separate from baseline evaluation.Source: §4 · p. 6
  • Four open-source models spanning 3B–8B parameters are fine-tuned to generate scores and rationales aligned with expert judgments, using GRPO reinforcement learning and LoRA.The reward combines prediction accuracy and structured-output format; training uses five epochs and a learning rate of 1e-5.Source: §4 · p. 6

5 EXPERIMENTS

VISJUDGE outperforms tested baselines in alignment with human ratings, while experiments show persistent scoring biases and declining model performance as visualizations become more complex. Tests in generation and recommendation systems also report practical gains from using VISJUDGE.

5.1 EXPERIMENTAL SETTINGS

  • The study evaluates 12 representative MLLMs on 648 balanced test samples using 1-to-5 scores, comparing Pearson correlation, MAE, and MSE against human ratings.Each model is run three times and scores are averaged; results are also examined by evaluation dimension and for systematic score-distribution biases.Source: §5.1 · pp. 6–7

5.2.1 CAN MLLMS ASSESS VISUALIZATION AESTHETICS AND QUALITY LIKE HUMANS?

  • MAE 0.421 and correlation 0.687 make VISJUDGE the best-performing model, with 23.9% lower MAE than GPT-5 and 42.5% higher correlation than GPT-4o.Domain-specific fine-tuning also reduced error across Qwen2.5-VL-3B, InternVL3-8B, and Llava-v1.6-mistral-7B backbones.Source: Table 3 · p. 7, §5.2.1 · p. 8
  • Across dimensions, MLLMs perform relatively well on Fidelity, moderately on Expressiveness, and worst on Aesthetics, whose average MAE is about 0.755.Aesthetic sub-dimension correlations range from 0.177 to 0.408; Qwen2.5-VL-72B-Instruct is the strongest open-source model overall (MAE 0.702, Corr. 0.440).Source: §5.2.1 · pp. 7–8

5.2.2 DO MLLMS EXHIBIT HUMAN-LIKE SCORING BEHAVIORS?

  • Most models inflate scores relative to human ratings, while Gemini-2.5-Pro is conservative; VISJUDGE’s mean score of 3.11 is close to the human mean of 3.13.Several models also concentrate scores around 4.0, whereas VISJUDGE is reported to maintain a broader, more balanced distribution.Source: §5.2.2 · p. 8

5.2.3 HOW DOES VISUALIZATION COMPLEXITY AFFECT MODEL PERFORMANCE?

  • Model-human correlation declines from single visualizations to multiple visualizations to dashboards, but VISJUDGE leads across all three types.Its correlations are 0.577, 0.565, and 0.375, respectively; most baselines drop substantially on dashboards, and some have negative Data Fidelity correlations.Source: §5.2.3 · pp. 8–9

5.2.4 HOW DO MODEL EVALUATION BEHAVIORS DIFFER IN PRACTICE?

  • For a chaotic treemap rated 1.67 by humans, Qwen2.5-VL-7B-Instruct scores 3.67 and Claude-4-Sonnet 3.08, while VISJUDGE scores 2.00 and identifies the chaotic layout.Source: §5.2.4 · p. 9

5.2.5 CAN VISJUDGE GENERALIZE TO REAL-WORLD APPLICATIONS?

  • In real-world applications, VISJUDGE feedback improves visualization generation by 6.07 points on average and raises recommendation Hit@1 and Hit@3 metrics.For recommendation, reported gains include +4.40% Data Queries Hit@1, +2.50% Overall Hit@1, and +5.30% Overall Hit@3; untuned base models can degrade generation performance.Source: §5.2.5 · p. 10

6 CONCLUSION

The paper introduces VISJUDGE-BENCH and VISJUDGE to assess and improve MLLM visualization evaluation. Results show gaps between existing models and human experts, while domain-specific training improves alignment.

6 CONCLUSION

  • VISJUDGE reduces MAE by 23.9% and improves correlation by 60.5% over GPT-5, mitigating gaps with human experts and scoring bias.The authors report that VISJUDGE-BENCH offers a standardized evaluation platform and VISJUDGE demonstrates domain-specific training as a viable approach.Source: §6 · p. 10
  • The paper constructs VISJUDGE-BENCH and fine-tunes VISJUDGE to validate domain-specific training for visualization quality assessment.Source: §6 · p. 10
  • The authors state that this work supports future research on finer evaluation and higher-quality visualization generation.Source: §6 · p. 10

7 LIMITATIONS AND FUTURE WORK

The benchmark’s scope and scoring approach leave several limits on what its evaluations can verify and represent.

7 LIMITATIONS AND FUTURE WORK

  • The benchmark focuses on static visualizations, with limited coverage of dynamic and interactive content.Source: §7 · p. 10
  • Web-collected samples typically lack raw data, so evaluation relies on visual presentation and cannot fully verify data fidelity.Source: §7 · p. 10
  • Point-estimate scores may not fully capture the diversity of human preferences.The authors propose exploring distribution-based evaluation methods.Source: §7 · p. 10

ETHICS STATEMENT

The authors anticipate no direct negative social impacts but identify ethical risks and safeguards relevant to use and benchmark construction.

ETHICS STATEMENT

  • The authors expect no direct negative social impacts, while urging caution and ethical oversight in sensitive domains or with potentially harmful models.Source: ETHICS STATEMENT · p. 11
  • The base models and benchmark datasets may inadvertently reflect biases, motivating future study of fairness across populations, cultures, and visualization styles.Source: ETHICS STATEMENT · p. 11
  • The authors report copyright compliance, fair and voluntary expert compensation, stereotype avoidance, and open-source release among their ethical considerations.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

This section documents the resources, procedures, and dataset-construction choices supporting reproducibility, including VisJudge-Bench’s coverage and composition and the use of language models in preparing the work. It characterizes the benchmark’s visualization types, styles, domains, and quality distributions, including variation across visualization formats and aesthetic dimensions.

REPRODUCIBILITY STATEMENT

  • The authors document dataset construction and annotation in Appendix A, the six-dimensional evaluation framework in Appendix C, and VisJudge implementation in Appendix D.They also report documenting experimental configurations, MAE, MSE, Pearson correlation, and human-consistency analysis methods; the dataset, scores, images, and evaluation prompts are released on GitHub.Source: REPRODUCIBILITY STATEMENT · p. 11

LLM USAGE STATEMENT

  • Claude-4-Sonnet assisted with English grammar and figure-design suggestions, while GPT-4o supported adaptive question generation and chart-metadata extraction during dataset construction.The authors state that all code was written, reviewed, and verified by them; prompts contained no private or sensitive data, and the authors take responsibility for the content.Source: LLM USAGE STATEMENT · p. 17

A.1 DATA COLLECTION AND CONSTRUCTION PIPELINE

  • The automated construction pipeline combines heuristic pre-screening, perceptual-hash deduplication, and AI-driven semantic refinement to build a structured dataset from web-scale data.It is intended to reduce manual annotation burden while maintaining the relevance and quality of collected visualizations.Source: §A.1 · pp. 17, 18, 20

A.2.1 OVERALL DATASET STATISTICS AND DISTRIBUTION

  • 3,090 expert-assessed visualizations have six-dimensional annotations and quality scores spanning 1.00–4.89 (mean 3.13, std 0.72).The dataset is designed to cover visualization types, evaluation dimensions, and quality levels across business intelligence, academic research, and data journalism.Source: §A.2.1 · p. 20

A.2.2 VISUALIZATION TYPE AND SUBTYPE ANALYSIS

  • 1,041 single visualizations (33.7%), 1,024 multiple visualizations (33.1%), and 1,025 dashboards (33.2%) are organized into 22, 5, and 5 subtypes, respectively.The taxonomy is based on structural complexity and functional purpose and spans basic charts through advanced analytical dashboards.Source: §A.2.2 · p. 20
  • Basic bar, pie, line, and area charts make up 46.1% (480 samples) of single visualizations, while specialized forms account for 15.9% (165 samples).A second tier of analytic charts contributes 38.0% (396 samples); the remaining specialized forms include network graphs and map-based views.Source: §A.2.2 · p. 20
  • Comparison views make up 65.4% of multi-visualization samples, followed by small multiples (19.0%) and coordinated views (9.5%), reflecting common multi-view configurations (Chen et al., 2020).Among dashboards, analytical dashboards account for 72.5%; their prevalence mirrors genre analyses and design surveys (Bach et al., 2023).Source: §A.2.2 · pp. 20–22

A.2.3 DESIGN STYLE DISTRIBUTION ANALYSIS

  • Manual inspection classified styles as Professional/Standard (53.3%), Creative/Infographic (28.4%), or Minimalist (18.3%), but the coarse, subjective scheme does not capture style’s full context dependence.The design-style mean was 2.97, and its distribution covered low to high innovation; the authors describe this classification as preliminary and note that systematic classification requires dedicated annotation.Source: §A.2.3 · p. 22

A.2.4 APPLICATION DOMAIN DISTRIBUTION ANALYSIS

  • Economy, business and finance, science and technology, and labor account for 62.3% of domain labels; dashboard data are even more concentrated, with these domains comprising 78.5%.Domain co-occurrences include economy with labor, science and technology, and society; the patterns indicate that many visualizations address cross-domain topics.Source: §A.2.4 · p. 23

A.2.5 QUALITY SCORE DISTRIBUTION

  • Dashboards scored higher (mean: 3.555, median: 3.610) than single visualizations (mean: 2.910) and multiple visualizations (mean: 2.917), whose quality distributions were broad.The overall quality distribution was near-normal, with scores predominantly from 2.0 to 4.0; the dashboard distribution was right-skewed and concentrated at higher scores.Source: Figure 8 · p. 21, §A.2.5 · pp. 23–24
  • Across six evaluation dimensions, fidelity and expressiveness centered near 3, while color harmony and composition scored highest and design style lowest.Data fidelity had mean 3.138 and median 3.000; semantic readability and insight discovery each had mean 2.983 and median 3.000. Color harmony scored mean 3.369 and median 3.500, visual composition 3.314 and 3.500, and design style 2.972 and 3.000.Source: Figure 10 · p. 23, §A.2.5 · p. 24

A.2.6 LOW-QUALITY SAMPLE DISTRIBUTION AND DESIGN FLAW ANALYSIS

This section examines how visualization-quality failures vary across single views, multi-views, and dashboards, and details test-set composition and annotation procedures, including quality control and reliability. It reports differences in failure patterns by visualization type and expert review of the annotation process as rigorous and reliable, with most samples not requiring removal of abnormal ratings.

A.2.6 LOW-QUALITY SAMPLE DISTRIBUTION AND DESIGN FLAW ANALYSIS

  • 12.5% of single views, 9.2% of multi-views, and 1.3% of dashboards scored below 2, with dashboards described as more mature and engineered.The counts were 130/1041, 94/1024, and 13/1025, respectively.Source: §A.2.6 · p. 24
  • Single-view failures cluster in complex or uncommon charts; multi-view weaknesses center on semantic integration, while dashboard failures concentrate in analytical and interactive cases with coupled functional deficits.For single views, five chart categories accounted for 57.7% of low-quality cases; for comparison views, about 20% failed on data fidelity, semantic readability, and insight discovery simultaneously.Source: §A.2.6 · pp. 24–25

A.2.7 TEST SET DISTRIBUTION ANALYSIS

  • The 648-sample test set preserves the dataset’s broad mix of visualization types, subtypes, and application domains, with 7.4% low-quality and 11.6% high-quality samples.Its overall score mean was 3.13 (SD 0.72), spanning 1.11–4.89; the score distribution was intended to cover the full quality range.Source: §A.2.7 · pp. 25–26, Table 6 · p. 26

A.3.2 ANNOTATION TASK DESIGN / A.3.3 ANNOTATION INTERFACE AND WORKFLOW

  • Each of 3,090 visualizations received six 1–5 dimension ratings, with three annotators independently assessing each image in balanced batches and using a chart-specific interface.Tasks presented 15 images and 90 questions, typically taking 30–60 minutes; the interface required answers to all six questions and allowed confidence ratings and optional uncertainty flags.Source: §A.3.2 · p. 27, §A.3.3 · p. 28

A.3.4 CROWDSOURCING QUALITY CONTROL AND CANDIDATE STRATEGY GENERATION / A.3.5 EXPERT INTERFACE AND ANNOTATION

  • Quality control combined obvious-pair attention checks with candidate strategies for high-disagreement samples, including outlier removal, abnormal-score filtering, and dimension-specific bias flags.Samples with annotator-score standard deviation above 1.0 were flagged for further analysis; the expert interface presented candidate signals as support rather than a replacement for judgment.Source: §A.3.4 · pp. 28–29, §A.3.5 · p. 29

A.4.1 OVERALL ANNOTATION QUALITY ASSESSMENT

  • Crowd-crowd MAE was 0.6441 for dashboards, 0.6842 for single views, and 0.6963 for multi-views; crowd-expert MAE followed the same ordering at 0.5451, 0.5534, and 0.5854.Crowd-expert MAE was lower than crowd-crowd MAE for all three visualization types, which the authors interpret as evidence that expert ratings balance annotator perspectives.Source: §A.4.1 · pp. 29–30

A.4.2 ANNOTATION QUALITY ACROSS ANNOTATOR BACKGROUNDS

  • Professional annotators had within-group MAE 0.4245 versus 0.4881 for non-professionals, while subgroup differences were limited and overall error patterns remained consistent across backgrounds.Doctoral annotators’ within-group MAE was 0.3944, but the authors caution that the doctoral group was small; they conclude that the main annotation findings are robust across backgrounds.Source: §A.4.2 · p. 30

A.4.3 EXPERT QUALITY CONTROL IMPACT ANALYSIS

  • Experts reviewed all 3,090 samples, adjusting about 11% at the overall evaluation level and applying sub-dimension bias correction to 16%.The reported overall adjustments used the malicious scoring filter for 7% of samples and outlier removal for 4%.Source: §A.4.3 · p. 30
  • Expert review found quality control rigorous and reliable, with high overall crowd-annotation quality and most samples not requiring removal of abnormal ratings.Source: §A.4.3 · p. 31

A.4.4 DIMENSION-SPECIFIC QUALITY ANALYSIS

The analysis examines how expert adjustments vary across quality dimensions and visualization types, then uses score-based and dimension-specific cases to assess human–VISJUDGE alignment and characterize model errors.

A.4.4 DIMENSION-SPECIFIC QUALITY ANALYSIS

  • Among 511 adjusted samples, semantic readability (19.20%), data fidelity (18.04%), and insight discovery (16.79%) were most frequently corrected.Experts adjusted objectively assessable dimensions more often, while making fewer adjustments to dimensions with more subjective aesthetic judgments.Source: §A.4.4 · p. 31
  • Adjustment patterns varied by format: fidelity led for single views (20.06%), semantic readability for multiple views (20.46%), while dashboards were more balanced.Source: §A.4.4 · p. 31
  • The findings support stricter control of objectively verifiable dimensions while preserving tolerance for subjective aesthetics, balancing annotation consistency with preference diversity.Source: §A.4.4 · p. 31

B.1 HIGH-SCORE CASE STUDIES: HUMAN-MODEL ALIGNMENT

  • In a representative high-score case, VISJUDGE's evaluation was largely consistent with human judgment when the visualization followed established best practices.Source: §B.1 · p. 34

B.2 MEDIUM-SCORE CASE STUDIES: HUMAN-MODEL ALIGNMENT

  • A representative medium-score visualization served its communicative purpose but did not meet high-quality standards across every dimension.Source: §B.2 · p. 34

B.3 LOW-SCORE CASE STUDIES: HUMAN-MODEL ALIGNMENT

  • In a low-score case, human ratings and VISJUDGE outputs both highlighted serious problems that compromised visualization effectiveness.Source: §B.3 · p. 34

B.4 DIMENSION-SPECIFIC CASE STUDIES: VALIDATING EVALUATION CRITERIA

  • Dimension-specific examples across fidelity, expressiveness, and aesthetics showed VISJUDGE evaluations aligned with human ratings on the intended criteria.Source: §B.4 · p. 34

B.5 MODEL ERROR ANALYSIS CASES

  • Base-model errors on low-quality charts reveal systematic misalignment and underscore the need to evaluate fidelity, expressiveness, and aesthetics explicitly.Source: §B.5 · p. 34

B.6 ERROR ANALYSIS OF MULTIPLE VISUALIZATIONS AND DASHBOARDS

The analysis finds that current multimodal models struggle more as visualizations become complex, with errors spanning aesthetic judgments, data fidelity, and dashboard semantics. It attributes these failures to reliance on local cues and weak cross-view or narrative understanding.

B.6 ERROR ANALYSIS OF MULTIPLE VISUALIZATIONS AND DASHBOARDS

  • As complexity rises from single charts to multi-view compositions and dashboards, model performance declines; analysis of 139 multi-visualization sets and 94 dashboard cases links errors to missing cross-view understanding.Models often treat views separately and rely on local cues instead of constructing cross-view semantic connections and analytical pathways.Source: §B.6 · p. 35
  • Models overrate regular grid layouts and coordinated styling while sometimes penalizing practical design trade-offs, indicating surface-level pattern matching over functional intent.In multi-visualization cases, three models rated visual composition above human experts in roughly 90% of cases, inflating scores by over 1.1 points on average.Source: §B.6 · p. 35
  • Models can mistake structural completeness for data validity, assigning high fidelity to charts without data marks and penalizing simplified schematic views.In multi-visualization cases, Qwen2.5-VL and InternVL3 overestimated fidelity in 95.0% and 96.4% of cases, respectively.Source: §B.6 · p. 35
  • On dashboards, models may reward visual polish despite weak logical coherence, or misread dense, information-rich displays as clutter or distortion.The analysis identifies weak understanding of narrative logic as a limitation in distinguishing polished artifacts from functionally effective analytical tools.Source: §B.6 · p. 35

C EVALUATION FRAMEWORK AND CRITERIA

The evaluation framework operationalizes Fidelity, Expressiveness, and Aesthetics through six dimensions with visualization-specific 1–5 scoring criteria. It applies these rubrics to assess single and multiple visualizations, requesting metric scores, reasoning, and an overall average in a consistent format.

C.1 DETAILED EVALUATION QUESTIONS AND SCORING CRITERIA

  • The framework operationalizes Fidelity, Expressiveness, and Aesthetics as six dimensions: data fidelity, semantic readability, insight discovery, design style, visual composition, and color harmony.These dimensions assess accuracy, communication, meaningful pattern discovery, innovation, spatial organization, and color coordination, respectively.Source: §C.1 · p. 46

1. Carefully observe the provided {image_type} type visualization chart

  • The rewriting prompt turns generic metric definitions into chart-specific questions and custom 1–5 scoring criteria grounded in observed chart features and content.Questions should identify specific features, address the metric, and end by requesting a 1–5 score; score descriptions reference the chart, with 1 and 5 representing worst-case and ideal implementations.Source: §C.1 · p. 46, §1 · pp. 46–47

C.1.1 SINGLE VISUALIZATION EVALUATION CRITERIA

  • Single-visualization rubrics assess fidelity, readability, insight, style, and composition, with scores progressing from misleading or unclear designs to accurate, understandable, useful, and well-organized charts.Fidelity considers axes, baselines, scales, encoding-label consistency, and distortion; insight scores distinguish trivial findings from useful patterns, trends, or anomalies.Source: §C.1.1 · pp. 48–50
  • The single-visualization rubric assesses spatial layout and color harmony, from uneven spacing and visual fatigue to coordinated layouts and readable, restrained palettes.Color scores range from chaotic choices that hinder information delivery to typically 1–3 coordinated main colors; the supplied spatial-layout excerpts describe levels 3–5.Source: §C.1.1 · pp. 50–51

C.1.2 MULTIPLE VISUALIZATION EVALUATION CRITERIA

  • Multiple-visualization data fidelity checks accurate axes, scales, labels, and cross-view consistency while penalizing distortion, cropping, and missing elements.The rubric spans severe misrepresentation or unreadable views to complete, clear views with coherent cross-view comparability.Source: §C.1.2 · p. 51
  • Semantic readability rates whether chart elements and encodings communicate information clearly, from unintelligible meanings to barrier-free understanding.The rubric considers whether users can understand encoding meanings and chart content, including the clarity of explanations and labels.Source: §C.1.2 · p. 52
  • Insight discovery evaluates whether views work together to reveal meaningful patterns, correlations, anomalies, and practically valuable findings.Higher scores represent stronger synergy across views and more significant comprehensive insights.Source: §C.1.2 · p. 52
  • Design style scores innovation, uniqueness, and coordinated stylistic language across views, from conventional designs to unified, original work.The rubric distinguishes originality while also considering whether views maintain a unified style.Source: §C.1.2 · pp. 52–53
  • Visual composition evaluates sub-view layouts and the overall arrangement, including proportions, spacing, and use of space.Scores range from chaotic layouts and unbalanced proportions to orderly arrangements with coordinated sizing and efficient space use.Source: §C.1.2 · p. 53
  • Multiple-visualization color harmony assesses appropriateness within views and coordination between them, including overall tone, color quantity, and saturation.The rubric description covers both sub-view combinations and consistency across the overall visualization.Source: §C.1.2 · pp. 53–54
  • Multiple-visualization color criteria progress from acceptable palettes with possible tone or saturation issues to highly coordinated, unified colors with well-controlled color quantity and saturation.The criteria also consider tone matching across views.Source: §C.1.2 · p. 54

C.1.3 DASHBOARD EVALUATION CRITERIA

  • Dashboard data-fidelity criteria assess whether components and the overall composition use justified axes and scales, consistent encodings and labels, and avoid distortion or cropping.The 1–5 rubric ranges from severe misrepresentation or unreadable key elements to complete, clear components with consistent comparisons across related metrics.Source: §C.1.3 · p. 54
  • Dashboard semantic-readability criteria assess whether visual encodings and indicator explanations let users understand component meanings and overall business information.The rubric ranges from unclear meanings and extensive guessing to clear interpretations and barrier-free understanding.Source: §C.1.3 · p. 55
  • Dashboard insight and style criteria assess business-value discovery and the prominence of key indicators, alongside design innovation, uniqueness, professionalism, and business aesthetics.The highest scores describe readily discoverable valuable trends, anomalies, or opportunities and highly original, professional design.Source: §C.1.3 · pp. 55–56
  • Dashboard composition and color criteria assess layout proportions, alignment, spacing, and information density, as well as component color appropriateness, palette coordination, tone, and saturation.The strongest rubric descriptions call for a harmonious, beautiful layout with efficient space use and coordinated colors suited to professional business aesthetics.Source: §C.1.3 · pp. 56–57

C.2 EVALUATION PROMPT TEMPLATES

  • The evaluation prompt maps Fidelity, Expressiveness, and Aesthetics to data fidelity, semantic readability, insight discovery, design style, visual composition, and color harmony.Its fields insert chart metadata and the rewritten criteria for the relevant dimensions.Source: §C.2 · p. 57
  • The prompt specifies total and customized criterion counts and supplies chart metadata, including chart type and design structure.Customized scoring guidelines are tailored to the chart.Source: §C.2 · p. 57
  • The criteria group evaluation into fidelity (data accuracy and truthfulness), expressiveness (information clarity and understandability), and aesthetics (visual aesthetics and refinement).Source: §C.2 · p. 58
  • The six scored metrics are data_fidelity, semantic_readability, insight_discovery, design_style, visual_composition, and color_harmony.Source: §C.2 · p. 58
  • Each metric receives an integer score from 1 to 5 with reasoning, and average_score is the mean of the six scores rounded to two decimal places.The response must contain only the JSON object.Source: §C.2 · p. 58

D MODEL IMPLEMENTATION AND TRAINING DETAILS

VISJUDGE uses a weighted accuracy-and-format reward and is trained with parameter-efficient fine-tuning; its performance improves with more training data. The supplied passages also describe evaluations in visualization generation and recommendation systems.

D.2 REWARD FUNCTION

  • The composite reward weights accuracy at 0.9 and output format at 0.1, rewarding proximity to human scores and complete, parsable JSON.The accuracy reward averages rewards across six dimensions and the overall average score; the format reward is 1.0 when all required score and reasoning fields are present, otherwise 0.Source: §D.2 · p. 59

D.1 SOFTWARE ENVIRONMENT

  • Training used SWIFT with PyTorch and DeepSpeed ZeRO Stage 2, and four open-source multimodal models were fine-tuned using LoRA and GRPO.LoRA rank and alpha were 128; training ran for 5 epochs at learning rate 1e-5, with global batch size 16 across 4 GPUs and bfloat16 mixed precision.Source: §D.3 · p. 59, §D.1 · p. 59

D.3 HYPERPARAMETER SETTINGS

  • The four fine-tuned models shared hyperparameter configurations, including LoRA on all linear layers and GRPO with beta 0.01.The configurations also used a cosine annealing scheduler with warmup ratio 0.1 and AdamW with weight decay 0.01.Source: §D.3 · p. 59

E.1 HOW DOES TRAINING DATA SCALE AFFECT MODEL PERFORMANCE?

  • Data samples for the scaling analysis were proportionally selected by visualization type and score distribution, with models trained for a single epoch.The study presents Figure 27 as the performance comparison across training data scales.Source: §E.1 · p. 60, Figure 27 · p. 60
  • Correlation rose from 0.341 to 0.639 as training data increased to 2,442 samples, while MAE fell from 0.847 to 0.440 and MSE from 1.045 to 0.309.The 500–1,000 sample range contributed 41.7% of total correlation gains; returns diminished beyond 1,000 samples but remained valuable.Source: §E.1 · p. 60, Figure 27 · p. 60

E.2 GENERALIZATION TO REAL-WORLD APPLICATIONS

  • VISJUDGE was evaluated as feedback in MatPlotAgent generation and as a reward model for reranking HAIChart recommendations under distribution shifts from training data.MatPlotAgent outputs were scored by GPT-4o on a 0–100 scale; HAIChart was evaluated on VizML using Hit@1 and Hit@3 for data queries, design choices, and overall recommendations.Source: §E.2 · p. 60, §E.2.1 · p. 60

E.2.2 RESULTS AND ANALYSIS

VISJUDGE improves visualization-generation and recommendation outcomes, and its fine-tuning gains extend across model architectures and out-of-distribution applications. The authors argue these results support its practical utility in visualization systems.

E.2.2 RESULTS AND ANALYSIS

  • VISJUDGE raised MatPlotAgent quality by an average of +6.07 points (range: +5.53 to +8.91), while 7B and 72B base models fell by -2.39 and -0.61 points.The reported base-model degradation is attributed to score inflation and concentration around 4.0, which limit discrimination and can misdirect generation feedback.Source: §E.2.2 · pp. 60–61, Table 7 · p. 61
  • Data Queries Hit@1 rose from 79.30% to 83.70% (+4.40%), Overall Hit@1 from 36.90% to 39.40% (+2.50%), and Overall Hit@3 from 67.40% to 72.70% (+5.30%).The gains were reported for HAIChart recommendation integration on VizML data, a distribution shift from the real Plotly community visualizations used for training.Source: §E.2.2 · p. 61, Table 8 · p. 61

E.2.3 CROSS-ARCHITECTURE GENERALIZATION

  • Fine-tuning improved correlation across four architectures by an average of +0.334 (+134.3% relative); LLaVA-1.6-mistral-7B rose from 0.180 to 0.605 (+236.1%).Improvements spanned Qwen, InternVL, and LLaVA models with parameter scales from 3B to 8B; weaker initial performance corresponded to larger relative gains.Source: §E.2.3 · p. 61

E.2.4 DISCUSSION

  • VISJUDGE outperformed 7B and 72B base models in both applications, with strong out-of-distribution performance on MatPlotAgent visualizations and HAIChart Plotly data.The authors interpret this as evidence of generalizable quality-assessment principles rather than memorization of training patterns.Source: §E.2.4 · p. 62
  • Consistent gains across architectures and generation and recommendation applications support VISJUDGE’s practical utility and readiness for production visualization systems.The authors also identify VISJUDGE-BENCH as a resource for developing robust, deployable visualization-assessment capabilities.Source: §E.2.4 · p. 62

Related papers

Loading 2510.22373v3…