PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptually Aligned Evaluation of Style Transfer

Yang Deng, Eleftherios Ioannou, David Mould, Steve Maddock, Paul L. Rosin, Yu-Kun Lai

arXiv:2610.10003v1cs.CV

TL;DR

Style-transfer evaluation lacks well-defined ground truth and standardised, perceptually aligned test data. ASTRA combines a human-judged benchmark with a learned evaluator, which achieves substantially greater alignment with human judgments than prior automated methods.

  • Problem

    Style transfer lacks well-defined ground truth and standardised, perceptually aligned evaluation data, while existing metrics may misclassify legitimate stylistic transformations.

    Source: §1 · p. 1

  • Method

    ASTRA combines a benchmark of content and style images, transfer results, and pairwise human judgments with a network that predicts scores from image triplets.

    Source: §1 · p. 2, §6 · pp. 10–11

  • Results

    ASTRA-Score achieves substantially greater alignment with human judgment than prior automated methods in a leave-one-out study.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ASTRA-Data and ASTRA-Score provide a standardised testbed for benchmarking style-transfer methods by alignment with human rankings.

    Source: §6 · p. 11

    The benchmark excludes many art media and styles beyond paintings.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Style transfer lacks a reliable evaluation standard: ground truth is inherently ill-defined, and existing automatic metrics often fail to reflect human preference. This paper introduces ASTRA (Assessment of Style TRansfer Algorithms), an approach for automatic evaluation of style transfer algorithms; it contains two components, ASTRA-Data and ASTRA-Score. ASTRA-Data consists of a benchmark image set of content and style references, a collection of style transfer results generated on the benchmark set, and user study data capturing human judgements through a two-stage pairwise comparison protocol. From these annotations, we derive ranking-based ground truth for content preservation, style fidelity, and overall preference. Based on ASTRA-Data, we construct ASTRA-Score, a learnt evaluator that predicts preference-aligned scores from content-style-stylization image triplets, enabling automatic and scalable evaluation of new models applied to the benchmark set. Experimental results demonstrate that ASTRA-Score achieves substantially higher correlation with human rankings compared to prior metrics. Overall, ASTRA establishes a robust mechanism for standardised evaluation of style transfer methods.

1. Introduction

Style-transfer evaluation is difficult because there is no well-defined ground truth and existing metrics can misjudge legitimate stylistic transformations. ASTRA combines a human-judged benchmark with a learned evaluator, which aligns more strongly with human judgments than prior automated methods.

1. Introduction

  • Style-transfer evaluation lacks well-defined ground truth, and existing metrics may treat legitimate stylistic transformations as errors.Artistic styles can involve structural abstraction or reconfiguration, while user studies are costly, difficult to scale, and hard to reproduce.Source: §1 · p. 1
  • The field lacks standardised, perceptually aligned evaluation frameworks and test data for artistic style transfer.Existing benchmarks often provide image collections without assessment protocols.Source: §1 · p. 1
  • ASTRA-Data was built from style and content images combined with varied transfer algorithms, producing results for a large-scale pairwise user study.Participants judged style capture, content preservation, and overall transfer satisfaction rather than general aesthetic appeal.Source: §1 · p. 2
  • ASTRA-Score achieves substantially greater alignment with human judgment than ArtFID, SRQE [CSC∗22], and Gram Loss in a leave-one-out study.It predicts stylisation effectiveness from a style image, content image, and transfer result.Source: §1 · p. 2
  • ASTRA pairs ASTRA-Data, a benchmark of images, transfer results, and human judgments, with ASTRA-Score, a network that assesses results on the benchmark set.The two components provide a benchmark and an automated assessment protocol for style-transfer algorithms.Source: §1 · pp. 1–2

2. Related Work

Prior work has used selected image pairs, general-purpose datasets, user studies, and automatic metrics to assess style transfer. The section describes remaining limitations in dataset design and in how evaluation methods align with human judgment.

2. Related Work

  • Evaluation components are often designed independently, contributing to limited reproducibility and inconsistent comparisons across methods.The section identifies a lack of a unified approach combining evaluation data, human judgments, and quantitative metrics.Source: §2 · p. 2

2.1. Evaluation Dataset for Style Transfer.

  • Earlier evaluations often used manually selected pairs or general-purpose datasets, while dedicated benchmarks have not always provided perceptual assessment protocols.AST-IQAD uses controlled content–style pairs, but its fixed output pool can make pair-specific difficulty hard to separate from general method performance.Source: §2.1 · pp. 2–3, §2 · p. 2

2.2. Evaluation Methods

  • Existing evaluation methods include user studies and a range of automatic metrics, but learned evaluators often rely on proxy signals or limited annotations.The methods surveyed include perceptual similarity, distribution-based measures, stylisation metrics, vision-language models, and learned evaluators.Source: §2.2 · p. 3

3. Construction of ASTRA-Data

ASTRA-Data combines a deliberately limited set of style and content images with representative stylisation methods to make broad pairwise evaluation tractable. Its selection choices aim to cover painterly variation while keeping the benchmark manageable.

3. Construction of ASTRA-Data

  • ASTRA-Data contains 12 style references, 6 content images, and 12 methods, yielding 864 stylised outputs across all combinations.The set is small enough to support human evaluation of a substantial fraction of possible pairs and computation of global rankings.Source: §3 · p. 3

3.1. Style Image Selection

  • Style images were sampled across content, colourfulness, and detail to represent a broad range of painterly characteristics.The selection also considered historical and cultural range, recognisable stylistic markers, and varied difficulty; the benchmark excludes many non-painting media.Source: §3.1 · pp. 3–4

3.2. Content Image Selection

  • Six NPRgeneral content images were selected, with two images each for people, nature, and artificial categories.Selection aimed to maximise coverage of properties including colourfulness, contrast, and gradients.Source: §3.2 · p. 4

3.3. Method Selection

  • The final method set retains ten NST methods, ChatGPT, and handcrafted low-level image-processing methods after filtering for diversity.Pairwise clean-FID distances under the same content–style conditions were used to identify and exclude methods with low distinctiveness.Source: §3.3 · p. 4

4. Annotation and Preference Analysis

The two-stage study turns pairwise judgments into global preference scores and examines how people weigh style and content. Overall preference tracks style fidelity more closely, though the trade-off with content depends on the style reference.

4.1. Two-Stage Study Design and Interface

  • In Stage 1, 17,568 votes came from 122 participants; Stage 2 collected 17,040 votes from 142 participants, for 34,608 total votes.Stage 1 compared methods within the same content–style pair; Stage 2 compared across pairs to enable global rankings.Source: §4.1 · p. 5

4.2. User Study Protocol and Quality Control

  • Stage 1 agreement was stronger than Stage 2: Cohen’s κ was 0.547–0.663 versus 0.389–0.515 across the three judgment criteria.The lower agreement occurred in the more challenging cross-pair comparisons.Source: §4.2 · p. 5

4.3. Two-Stage Ranking Stability Simulation

  • Stage 1 ranking stability exceeded ρ = 0.99 at about 1.2×10^4 comparisons; Stage 2 reached ρ ≈ 0.95 at about 1.4 × 10^4 comparisons.The study collected 17,568 and 17,040 votes in Stages 1 and 2, respectively, within the reported stable regimes.Source: §4.3 · p. 5

4.4. Preference Score Estimation

  • Rank Centrality converts pairwise results into global rankings, using each image’s stationary probability as its preference score.The two study stages combine within-pair comparisons with cross-pair comparisons to support global rankings for overall, style, and content preferences.Source: §4.4 · p. 6

4.5. Preliminary Data Analysis

  • Top-ranked examples often show distinctive style cues, while bottom-ranked examples include severe content degradation or weak stylisation.Both recognisable content and salient style cues matter, with style cues playing a particularly prominent role.Source: Figure 3 · p. 6, §4.5 · p. 7
  • Style-specific trade-offs vary: abstract styles show an inverse content–style relationship, while some styles show positive correlation or no clear trend.Fig. 6(c) shows positive correlation and Fig. 6(d) no clear trend; the trade-off varies substantially across styles, while content-image trends are largely consistent.Source: Figure 6 · p. 7, §4.5 · p. 7
  • Gatys has the highest mean overall-preference score but substantial variation; AdaAttN [LLH∗21] and AdaIN are more stable yet fall short overall.USO [WHC∗25] and OmniStyle [WLL∗25] have narrow error bars but are not competitive in overall scores.Source: §4.5 · p. 7
  • 80.3% of 18,747 conflict comparisons followed style preference, versus 19.7% following content preference.Overall effectiveness also correlates strongly with style fidelity and shows little pattern with content preservation.Source: §4.5 · pp. 6–7, Figure 4 · p. 6

5. ASTRA-Score and Experiments

ASTRA-Score predicts human-aligned content, style, and overall preference scores from content–style–stylisation triplets. It aligns more strongly with human rankings than existing metrics across held-out methods and remains robust when content or style references are withheld.

5.1. Evaluation Model

  • ASTRA-Score uses three independent regression predictors for overall preference, content preservation, and style similarity, with a frozen pretrained backbone and lightweight task-specific heads.VGG11 provides the strongest overall alignment among the evaluated backbones and is adopted by default.Source: §5.1 · p. 8

5.2. Evaluation Methods Comparison

  • The primary leave-one-method-out protocol trains on 11 methods and predicts rankings for the held-out method across its 72 generated images, repeating this for all 12 methods.Leave-one-content-out and leave-one-style-out analyses additionally withhold references and show strong correlations on unseen references.Source: §5.2 · pp. 9–10
  • Under leave-one-content-out evaluation, ASTRA-Score reaches Spearman correlations of 0.745 for content, 0.814 for style, and 0.793 for overall preference.It outperforms the strongest baselines by 53.9% for style and 45.5% for overall preference; DINO scores 0.759 for content versus ASTRA-Score’s 0.745.Source: Table 3 · p. 9, §5.2 · p. 10
  • Across all evaluation heads, ASTRA-Score achieves substantially higher correlation with study-derived rankings than existing automatic metrics.Prior metrics show moderate or weak correlation, especially for style and overall evaluations.Source: §5.2 · pp. 9–10
  • Under leave-one-style-out evaluation, ASTRA-Score reaches Spearman correlations of 0.711 for content, 0.700 for style, and 0.658 for overall preference.The strongest baselines score 0.673, 0.522, and 0.525, respectively.Source: Table 4 · p. 10, §5.2 · p. 10

6. Conclusion

The conclusion presents ASTRA as a human-preference benchmark and learned evaluator for style transfer. Together, ASTRA-Data and ASTRA-Score support standardized evaluation on the benchmark image set.

6. Conclusion

  • ASTRA-Data derives ranking-based ground truth for content preservation, style similarity, and overall preference from pairwise human annotations.The paper describes the benchmark as representative yet tractable.Source: §6 · p. 10
  • ASTRA-Score predicts preference-aligned scores from content–style–stylisation triplets, providing scalable and consistent estimates on the benchmark image set.The appendix provides supplementary construction, user-study, analysis, and implementation details.Source: §6 · p. 11, (Appendix) · p. 14

Appendix A: Benchmark Data Selection and Generation Details

The appendix describes a benchmark built from diverse artistic styles and handcrafted LLIP approaches alongside learning-based methods. It details style-specific construction choices and supplementary user-study procedures.

Appendix A: Benchmark Data Selection and Generation Details

Hokusai (nature, low colour, low detail; Asia, 19th century)

  • Other selected references include abstract geometric motifs, fragmented spatial arrangements, detailed portraits, painterly landscapes, and deliberately distorted figures.These examples vary in colour, detail, and structure, including challenges to texture-dependent methods and clean separation of style from content.Source: Hokusai (nature, low colour, low… · pp. 14–15

Details of LLIP Method Construction

  • LLIP combines existing methods, hybrid pipelines, and custom procedures because no single model covers all styles.The approaches are designed to complement learning-based methods, particularly for styles that existing neural models struggle to capture.Source: Appendix A · p. 14, Details of LLIP Method Construction · p. 15

Design Strategies

  • LLIP stylisations use style-aligned method selection, combinations of methods, multi-style methods, or custom pipelines.The appendix gives examples for each strategy, including Kandinsky and Bierstadt, Rembrandt and Veronese, Aboriginal art and The Great Wave, and Mondrian and Picasso.Source: Design Strategies · p. 15

Style-Specific Implementations

  • Image analogies form the base for Aboriginal art and The Great Wave, while The Great Wave also uses colour transfer; other styles combine filtering, mapping, abstraction, or averaged outputs.The described examples include Rembrandt, Kandinsky, Bierstadt, Veronese, and Chinese landscape.Source: Style-Specific Implementations · p. 15
  • Custom pipelines tailor stylisations to particular visual structures: Carr uses segmented regions and painterly textures, while Mondrian builds a restricted-palette grid from rectangular blocks.Ptolemaic Egyptian and Picasso pipelines add layered colour, textures, line drawings, or structured patterns.Source: Style-Specific Implementations · pp. 15–16

Appendix B: User Study Protocol

  • The user-study protocol appendix covers participant training, quality control, and procedures intended to support reliable and consistent annotations.It also describes attention checks and fatigue mitigation.Source: Appendix B · p. 16

Training Phase

The protocol trains participants to judge content, style, and overall transfer quality, then applies response checks and reports task-dependent annotation agreement. It also notes that preference differences across demographic or cultural groups were not studied.

Training Phase

  • Participants received a tutorial defining content as the represented scene, style through artistic characteristics, and transfer as recreating one image’s content in another’s style.They were asked to judge the balance of content preservation and style similarity, not subjective appeal alone.Source: Training Phase · p. 16
  • Participants had to answer training questions correctly after immediate feedback before proceeding to the evaluation tasks.Separate illustrative examples with clearly distinguishable answers were also presented with feedback before the main study.Source: Training Phase · pp. 16–17

Quality Control

  • Stage 1 participants needed at least five of six attention checks correct, while Stage 2 participants needed at least three of four.A mandatory timed midpoint break was included to mitigate fatigue.Source: Quality Control · p. 17
  • After attention-check filtering, 264 of 407 submissions were retained; rejected submissions had a median 4.526 s per question versus 8.573 s for retained submissions.The authors report this response-time contrast as additional evidence that the checks identified inattentive or low-effort responses.Source: Quality Control · p. 17

Annotation Consistency

  • Duplicate-annotation agreement was higher in Stage 1 than Stage 2: κ was 0.663 versus 0.515 for content, 0.547 versus 0.389 for style, and 0.547 versus 0.416 for overall preference.Stage 2 comparisons cross different content–style pairs and require more demanding perceptual judgements.Source: Annotation Consistency · pp. 17–18
  • The study was not designed to analyse demographic or cultural differences, so preference stability across user groups remains future work.The authors characterize annotation consistency as meaningful but task-dependent despite the subjective evaluation setting.Source: Annotation Consistency · p. 18

Appendix C: Ranking Stability Simulation

Appendix C tests whether the two-stage pairwise-comparison study supports stable within-group and global rankings. The simulations and graph diagnostics indicate that the collected comparisons support global ranking estimation.

Appendix C: Ranking Stability Simulation

  • The theoretical reference scale is approximately 30 comparisons per group for Stage 1 and 5.8×10^3 for Stage 2.These are reference scales, not exact thresholds.Source: Appendix C · p. 18
  • Approximately 0.95 Spearman correlation is reached at 1.4×10^4 Stage 2 comparisons, with the study’s 17,040 votes in a highly correlated regime.The figure shows recovery improving as cross-group comparisons increase.Source: Figure 15 · p. 18, Appendix C · pp. 18–19
  • Stage 2 adds comparisons across content–style groups because Stage 1 alone cannot establish relative rankings between groups.The simulation assigns latent scores using method, content, and style effects with interaction noise, then aggregates comparisons using Rank Centrality.Source: Appendix C · p. 18
  • The analyses conclude that the two-stage study’s scale supports reliable preference modelling and global ranking estimation.The Stage 2 vote count falls in the regime where recovered rankings are highly correlated with ground truth.Source: Appendix C · p. 19

Comparison Graph and Rank Centrality Diagnostics

  • Combining both stages produces one connected comparison component across all 864 stylised images, with an average node degree of approximately 45.4.Stage 1 alone forms 72 disconnected components; Stage 2 links them, allowing a global stationary ranking.Source: Comparison Graph and Rank Centrality… · p. 19
  • No disconnected graph appeared in 106 simulations of Stage 2 sampling.The accompanying group-level connectivity analysis bounds disconnection probability using a union bound over possible subsets.Source: Comparison Graph and Rank Centrality… · p. 19
  • The Rank Centrality transition matrices have non-zero spectral gaps: 0.0199 for content preservation, 0.0421 for style similarity, and 0.1000 for overall preference.These diagnostics are presented alongside connectivity and ranking-recovery analyses.Source: Comparison Graph and Rank Centrality… · p. 19

Appendix D: ASTRA Analysis

Appendix D examines whether reference exposure is balanced across methods in Stage 2. The reported counts are approximately balanced for both style and content references.

Appendix D: ASTRA Analysis

  • The appendix frames its additional analyses as checks on sampling balance and as examinations of preference variation across content and style conditions.It presents these analyses as further evidence for benchmark robustness and style-dependent trade-offs.Source: Appendix D · p. 19

Exposure Statistics of Content and Style References

  • The analysis checks reference exposure by counting how often each style or content reference appears with each method.Tables 7 and 8 report these counts for Stage 2 comparisons.Source: Exposure Statistics of Content and… · p. 19
  • Across Stage 2 comparisons, each method appears roughly 200–270 times per style reference and approximately 420–530 times per content image.The authors state that this balance helps average out method-specific biases when pairwise judgements are aggregated into Rank Centrality scores.Source: Exposure Statistics of Content and… · p. 19

Additional Analysis of Content-Style Trade-off

The additional analyses examine how content and style win rates vary across individual styles and content images. They report that the content–style trade-off depends on the style, while highly ranked results generally balance the two objectives better.

Additional Analysis of Content-Style Trade-off

  • The per-style analysis plots each method’s content and style win rates for individual style references.Figure 16 presents the distributions across styles.Source: Additional Analysis of Content-Style… · p. 19
  • Abstract styles such as Mondrian and Kandinsky show an inverse relationship between content and style preferences, while only Ptolemaic Egyptian shows a positive correlation.For other styles, no clear trend emerges, so the trade-off is not universal and depends on style properties.Source: Additional Analysis of Content-Style… · p. 19
  • The per-content analysis groups results by input image and computes each method’s content and style win rates.Figure 17 presents distributions for all content images, with one subplot per content instance.Source: Additional Analysis of Content-Style… · p. 19
  • Figure 9 compares each method’s three highest- and three lowest-ranked stylised results by overall human preference.Source: Qualitative Comparison Across Methods · p. 19
  • Top-ranked results generally balance content preservation and stylistic expression better, while bottom-ranked results often show structural distortions or weak style transfer.The analysis also reports that some methods prioritize strong style at the cost of content fidelity, whereas others preserve structure but lack distinctive style.Source: Qualitative Comparison Across Methods · pp. 19–20

Appendix E: Training Details of Regression Heads

The regression heads are lightweight MLPs trained independently to predict human-preference-derived Rank Centrality scores. The appendix specifies their architecture and optimization settings.

Appendix E: Training Details of Regression Heads

  • Each independently trained regression head is a two-layer MLP with ReLU activations, dropout of 0.2, a hidden dimension of 256, and a scalar output.Training targets are human-preference-derived Rank Centrality scores.Source: Appendix E · p. 20
  • The heads use Smooth L1 loss, AdamW, and a learning rate of 2×10^-4.Source: Appendix E · p. 20
  • Training runs for 40 epochs with a batch size of 32.Source: Appendix E · p. 20

Appendix F: Backbone Architecture Ablation

The ablation compares frozen pretrained feature backbones for ASTRA-Score under leave-one-method-out evaluation. VGG11 leads across content, style, and overall preference, while greater backbone capacity does not consistently improve alignment.

Appendix F: Backbone Architecture Ablation

  • The comparison spans MobileNetV3, VGG, ResNet, ViT, and DINOv2, with pretrained backbones frozen and only task-specific regression heads trained.Table 9 reports mean and standard deviation of Spearman’s ρ and Kendall’s τ across held-out stylisation methods for the three criteria.Source: Appendix F · p. 20, Table 9 · p. 23
  • Greater backbone capacity does not consistently improve human-judgment alignment: VGG19, ViT-B/16, and DINOv2 do not outperform VGG11 consistently.ResNet50 is competitive for style similarity but weaker for content preservation and overall preference; VGG11 is the default feature extractor.Source: Appendix F · pp. 20–21
  • 0.718, 0.714, and 0.667 Spearman’s ρ make VGG11 strongest for content, style, and overall preference in LOMO.Its corresponding Kendall’s τ values are 0.540, 0.531, and 0.492.Source: Appendix F · p. 20, Table 9 · p. 23
Loading 2610.10003v1…