PaperlayerView exact revision on arXiv

Source-linked AI summary

UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark

Zhaokun Zhou, Qiulin Wang, Bin Lin, Yiwei Su, Rui Chen, Xin Tao, Amin Zheng, Li Yuan, Pengfei Wan, Di Zhang

arXiv:2404.09619v1cs.CVcs.AI

TL;DR

Traditional IAA methods are limited to individual data sources or tasks, despite the multi-stage nature of human aesthetic judgment. UNIAA combines an MLLM trained with converted IAA data and a benchmark spanning perception, description, and assessment; experiments report its effectiveness and the feasibility of MLLMs for aesthetic tasks. The authors also note that MLLMs remain short of senior-level human proficiency, especially in composition.

  • Problem

    Traditional IAA methods often focus on one data source or task, while datasets cover only parts of the human aesthetic process and lack standardized formats.

    Source: §1 · pp. 1–2

  • Method

    UNIAA fine-tunes an LLaVA-based MLLM on aesthetic instruction data converted from existing datasets and evaluates it with a benchmark spanning perception, description, and assessment.

    Source: §1 · pp. 2–3, §3 · p. 3

  • Results

    Experiments demonstrate UNIAA-LLaVA’s effectiveness, UNIAA-Bench’s authority, and the feasibility of MLLMs for aesthetic tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors state that dependable aesthetic descriptions could help alleviate human experts’ IAA workload.

    Source: §6 · p. 9

    MLLMs remain short of senior-level human proficiency, with composition the largest gap between MLLMs and humans.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

As an alternative to expensive expert evaluation, Image Aesthetic Assessment (IAA) stands out as a crucial task in computer vision. However, traditional IAA methods are typically constrained to a single data source or task, restricting the universality and broader application. In this work, to better align with human aesthetics, we propose a Unified Multi-modal Image Aesthetic Assessment (UNIAA) framework, including a Multi-modal Large Language Model (MLLM) named UNIAA-LLaVA and a comprehensive benchmark named UNIAA-Bench. We choose MLLMs with both visual perception and language ability for IAA and establish a low-cost paradigm for transforming the existing datasets into unified and high-quality visual instruction tuning data, from which the UNIAA-LLaVA is trained. To further evaluate the IAA capability of MLLMs, we construct the UNIAA-Bench, which consists of three aesthetic levels: Perception, Description, and Assessment. Extensive experiments validate the effectiveness and rationality of UNIAA. UNIAA-LLaVA achieves competitive performance on all levels of UNIAA-Bench, compared with existing MLLMs. Specifically, our model performs better than GPT-4V in aesthetic perception and even approaches the junior-level human. We find MLLMs have great potential in IAA, yet there remains plenty of room for further improvement. The UNIAA-LLaVA and UNIAA-Bench will be released.

1 INTRODUCTION

Existing IAA datasets and models cover only parts of aesthetic evaluation, while human aesthetic judgment involves multiple stages. UNIAA combines an MLLM baseline, low-cost dataset conversion, and a benchmark spanning perception, description, and assessment.

1 INTRODUCTION

  • Existing IAA datasets cover only parts of the human aesthetic process and use nonstandard annotation formats, while methods generally focus on single aesthetic tasks.Scoring models struggle to recognize aesthetic attributes, and description models cannot score photos.Source: §1 · pp. 1–2
  • The proposed framework uses LLaVA fine-tuned on aesthetic visual-instruction data, combining visual perception with the LLM’s aesthetic knowledge.The authors describe MLLMs as a natural fit because they bring visual perception and knowledge capacity to the task.Source: §1 · p. 2
  • IDCP converts existing aesthetic attributes and comments into MLLM visual-instruction-tuning data without requiring new manual annotation.The converted data is used to fine-tune UNIAA-LLaVA.Source: §1 · p. 2
  • UNIAA-Bench evaluates MLLMs across Aesthetic Perception, Aesthetic Description, and Aesthetic Assessment, while UNIAA-LLaVA unifies those tasks in one baseline.The benchmark includes in-domain and in-the-wild images; its perception, description, and assessment tests use questions, expert descriptions, and zero-shot scoring, respectively.Source: §1 · pp. 2–3
  • The authors report experiments demonstrating UNIAA-LLaVA’s effectiveness, UNIAA-Bench’s authority, and the feasibility of MLLMs for aesthetic tasks.The framework and benchmark are presented as aligning learning and evaluation with the visual aesthetic process of human experts.Source: §1 · pp. 2–3

2 RELATED WORK

Prior IAA work has developed scoring, attribute-recognition, and description methods and datasets, but most models address only one or two subtasks. MLLMs offer a language-mediated route to combining aesthetic information across tasks.

2.1 Image Aesthestic Assessment

  • IAA includes direct aesthetic scoring, attribute recognition, and aesthetic description, but most models handle only one or two subtasks.Existing datasets include scores, attributes, and descriptions, and their shared aesthetic knowledge offers potential for integrated learning.Source: §2.1 · p. 3
  • MOS-supervised learning may lack interpretability and overfit label noise, while dataset- and task-specific models limit general-purpose evaluation.The paper identifies separate training for perception, description, and assessment as a barrier to comprehensive learning and applicability.Source: §2.1 · p. 3

2.2 Multi-modality Large Language Models

  • MLLMs combine visual encoders with LLMs, and their language-mediated inputs and outputs can integrate aesthetic instruction tuning with multiple IAA subtasks.The authors’ goal is a unified aesthetic model that integrates aesthetic information across tasks and responds to open-ended queries.Source: §2.2 · p. 3

3 UNIAA-LLAVA

UNIAA-LLaVA is fine-tuned on instruction data converted from existing IAA datasets through IDCP, a pipeline of data selection, filtering, and question-answer generation. The section describes the data choices, conversion advantages, and model-training setup.

3 UNIAA-LLAVA

  • The authors start from LLaVA and instruction-fine-tune it on converted data from existing IAA datasets to create an IAA MLLM for perception, description, and assessment.They describe the data-conversion paradigm as a low-cost alternative to collecting manual annotations.Source: §3 · p. 3

3.1 IAA Datasets Conversion Paradigm

  • IDCP converts existing IAA annotations into instruction-tuning data through three steps: data selection, data filtering, and question-answer generation.The converted data is used to train UNIAA-LLaVA.Source: §3.1 · p. 3

3.1.1 Data Selection.

  • IDCP selects attributes and comments from five open-source datasets rather than scores, whose subjectivity and differing dataset scales make them difficult to unify.The selected sources include AVA, AADB, ICAA17K, PARA, and PCCD; attribute annotations support visual question answering, while comments convey aesthetic features and experiences.Source: §3.1.1 · pp. 3–4
  • Supervised training on specific datasets using UNIAA-LLaVA’s vision encoder achieves SOTA performance, while the authors also report good zero-shot scoring capability.The supervised experiment is presented as demonstrating the generalization of the learning framework.Source: §3.1.1 · p. 4

3.1.2 Data Filtration.

  • Data filtering checks annotation completeness and accuracy, removes overly simple or difficult examples, and balances aesthetic-quality ranges and answer options.The filtering process also balances yes/no responses.Source: §3.1.2 · p. 4

3.1.3 QA Generation.

  • The QA-generation pipeline expands expert-written attribute questions into varied forms and cleans aesthetic comments before matching them to questions.Attribute questions use yes/no, what, or how formats; comment cleaning uses GPT and LLaVA, and question matching considers comment length and dimensions.Source: §3.1.3 · pp. 4–5

3.1.4 IDCP Advantages Discussion.

  • IDCP is described as low-cost, uniform across comments and attributes, and scalable to newly available aesthetic datasets.The authors state that incorporating more data will further enhance model aesthetic capabilities.Source: §3.1.4 · p. 5

3.2 Image Aesthetic Visual Instruction Tuning

  • For aesthetic instruction tuning, the authors use an MLLM already fine-tuned in two stages and open vision-encoder parameter training to support finer visual perception.The two stages are visual-backbone/LLM representation alignment followed by visual instruction tuning.Source: §3.2 · p. 5

4 UNIAA-BENCH

UNIAA-Bench evaluates MLLMs across aesthetic perception, description, and assessment, using in-domain and in-the-wild resources. Its tasks range from attribute questions and expert descriptions to zero-shot image scoring.

4 UNIAA-BENCH

  • UNIAA-Bench evaluates MLLMs in aesthetic perception, description, and assessment, using both in-domain and in-the-wild images.Source: §4 · p. 5

4.1 Aesthetic Perception

  • UNIAA-QA contains 5,354 image-question-answer samples, with one multiple-choice question per image covering six aesthetic dimensions.Source: §4.1 · p. 5

4.2 Aesthetic Description

  • UNIAA-Describe pairs 501 images with expert-written, manually reviewed descriptions averaging 100 words, evaluated for completeness, preciseness, and relevance.Source: §4.2 · p. 6

4.3 Aesthetic Assessment

  • Assessment pools logits for five rating-level words into a weighted aesthetic score, then evaluates zero-shot scoring with SRCC and PLCC.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

UNIAA-LLaVA is evaluated across aesthetic perception, description, and assessment, with comparisons to MLLMs and human judgments. Results show strong perception and assessment performance, while description and human-level perception remain challenging.

5 EXPERIMENTS

  • The experiments assess 13 variants across 10 open-source MLLMs, GPT-4V, and Gemini-Pro-Vision, and report human perception judgments.The benchmark includes in-domain and in-the-wild evaluation splits.Source: §5 · p. 7

5.1 Results on Aesthetic Perception

  • With aesthetic fine-tuning data from the IAA conversion pipeline, UNIAA-LLaVA achieves 78.48% overall perception accuracy, exceeding GPT-4V’s 74.74%, and leads in four of six dimensions except composition and content.Compared with original LLaVA-1.5, its composition accuracy improves from less than 50% to 66.82%.Source: §5.1 · p. 7

5.2 Result on Aesthetic Description

  • GPT-4V has the highest overall description performance, while UNIAA-LLaVA leads in completeness and preciseness; all MLLMs fall short of 3.6/6.0 overall.The table reports no human results because of their high cost and difficulty.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Result on Aesthetic Assessment

  • UNIAA-LLaVA leads zero-shot aesthetic scoring on all four mainstream IAA datasets, including TAD66K and Flick-AES, despite not training on them or using MOS values.Its score-token predictions are evaluated against ground truth using PLCC/SRCC; supervised vision-encoder training also reaches PLCC 0.838 on AVA and 0.553 on TAD66K.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Ablation Study

  • Removing IDCP stages lowers performance; raw comments reduce WILD performance by 3.8%, while omitting description rewriting significantly harms description ability.Using only attributes or only descriptions also decreases perception, with the larger drop from descriptions.Source: §5.4.1 · p. 8, Table 4 · p. 8

5.4.2 UNIAA on mPLUG-OWL2.

  • UNIAA-mPLUG-OWL2 reaches 77.89% perception accuracy, below UNIAA-LLaVA but above GPT-4V, and improves description and assessment over its vanilla model.It is trained using the same data as UNIAA-LLaVA.Source: §5.4.2 · p. 8

5.4.3 Ablation of Fine-tuning the vision encoder.

  • Freezing the vision encoder lowers perception to 76.82% from 78.48% and similarly lowers assessment, but slightly improves aesthetic description.The comparison keeps other training settings consistent.Source: §5.4.3 · pp. 8–9

5.4.4 Generalization on image quality benchmark.

  • Without image-quality-dataset training, UNIAA-LLaVA surpasses many open-source MLLMs and models trained with Q-Instruct on Q-bench.The authors report consistency between Q-bench and UNIAA-Bench results.Source: §5.4.4 · p. 9, Table 5 · p. 9

6 CONCLUSION

UNIAA combines UNIAA-LLaVA, trained with the IDCP dataset-conversion paradigm, and UNIAA-Bench, which evaluates aesthetic perception, description, and assessment. The authors report stronger aesthetic abilities than other MLLMs, while noting a remaining gap to dependable general visual aesthetic assistance.

6 CONCLUSION

  • UNIAA-LLaVA exhibits superior aesthetic abilities to other MLLMs, supporting the effectiveness of IDCP.The authors also report that MLLMs remain a significant distance from becoming dependable general visual aesthetic assistants.Source: §6 · p. 9
  • UNIAA-Bench evaluates MLLMs on aesthetic-attribute perception, thorough description, and aesthetic-quality scoring.The benchmark is intended to examine aesthetic capabilities across these three aspects.Source: §6 · p. 9
  • UNIAA comprises UNIAA-LLaVA and UNIAA-Bench, with IDCP converting existing aesthetic datasets into visual-instruction-tuning data.The authors describe IDCP as a first-of-its-kind multimodal IAA dataset-conversion paradigm.Source: §6 · p. 9

A.1 Data Source Details

The appendix describes the annotation sources used in IDCP and illustrates how prompts and language models turn them into visual-instruction data. The process generates varied aesthetic questions and rewrites comments into focused aesthetic descriptions.

A.1 Data Source Details

  • 278,181 data sources are reported for IDCP, whose annotations include overall comments, photo style, binary attributes, attribute quality levels, single-attribute comments, and color schemes.The appendix describes these as data from five IAA dataset sources.Source: Table 6 · p. 11, §A.1 · p. 11

A.2 Prompt of Question Generation on Aesthetic Attributes

  • ChatGPT designs a question for an aesthetic dimension and rewrites it into five differently phrased questions with the same meaning.The example asks about the composition used in a photograph.Source: §A.2 · p. 11

A.3 Prompt of Description Generation

  • LLaVA generates image-content and spatial descriptions without aesthetic information, while GPT rewrites annotated comments using those captions.The rewrite filters out weakly aesthetic or personal comments, combines the remaining comments into a professional aesthetic comment, and follows comments over captions when they conflict.Source: §A.3 · pp. 11–12

A.4 Examples of IDCP Data

The examples show IDCP training data spanning aesthetic attributes, image-level comments, and camera-use evaluations. They pair varied aesthetic questions with answers expressed as labels, scores, or prose.

A.4 Examples of IDCP Data

  • IDCP examples pair attribute questions with categorical answers, including “Soft Focus,” “No” for balance, and “Complementary” for color scheme.Another example names orange and violet as the complementary colors.Source: §A.4 · p. 12
  • Aesthetic-comment examples turn image evaluations into extended critic-style prose, covering composition, color, lighting, appeal, and flaws.The examples include both AVA and PCCD comments.Source: §A.4 · p. 12
  • A camera-use example evaluates an image through the photographer’s angle, f-stop, and shutter-speed choices.Its answer describes the resulting perspective and image detail as clear.Source: §A.4 · p. 12

A.5 Hyper-parameters of Training

The appendix reports fine-tuning durations and the final aesthetic-instruction stage for UNIAA-LLaVA. Table 7 lists the tuning hyperparameters for UNIAA-LLaVA and UNIAA-mPLUG-OWL2.

A.5 Hyper-parameters of Training

  • UNIAA-LLaVA fine-tuning took 16 hours, while UNIAA-mPLUG-OWL2 took 22 hours using 8 NVIDIA V100 GPUs.The durations are reported for fine-tuning with the IAA instructions.Source: §A.5 · p. 12
  • UNIAA-LLaVA received a third fine-tuning stage using only converted aesthetic instruction-tuning data for 2 epochs.The stage used a learning rate of 2e-5 and batch size of 128.Source: §A.5 · p. 12
  • Table 7 lists hyperparameters for visual instruction tuning of UNIAA-LLaVA and UNIAA-mPLUG-OWL2, matching the original baselines.The accompanying text says the models were tuned with 8 NVIDIA V100 GPUs.Source: Table 7 · p. 12, §A.5 · p. 12

B DETAILS ON UNIAA-BENCH EVALUATION

The benchmark appendix illustrates its perception, description, and assessment tasks and documents evaluation prompts and response-handling rules.

B.1 Examples of UNIAA-QA

  • UNIAA-QA perception examples cover content and theme, composition, color, light, focus, and sentiment through natural-language questions with answer choices.The focus example asks for shot type, with close-up shown as a possible answer.Source: §B.1 · pp. 12–13

B.2 Example of UNIAA-Describe

  • The description example evaluates composition, color, lighting, focus, and improvement suggestions in a single detailed response.Its sample critiques empty space, low saturation, and insufficient brightness, then proposes changes to framing, color, and exposure.Source: §B.2 · p. 13

B.3 Example of UNIAA Assessment Dataset

  • The assessment examples draw on four datasets: AVA, PARA, TAD66K, and Flickr-aes, whose maximum scores are 10, 5, 10, and 10, respectively.The test sets contain 19,928 AVA, 14,079 PARA, 3,000 TAD66K, and 4,737 Flickr-aes samples.Source: §B.3 · p. 13, Figure 4 · p. 13

B.4 Evaluation Details for Aesthetic Perception

  • Most models receive multiple-choice perception prompts, including questions about image color saturation with designated correct and incorrect options.The example asks whether a picture is oversaturated, undersaturated, or appropriately saturated.Source: §B.4 · p. 13

B.5 Exceptions Perception Responses.

  • For response exceptions, InstructBLIP paragraphs are mapped to an option with ChatGPT, while nonexistent Qwen-VL options or insufficient-information answers count as wrong.GPT-4V accuracy is calculated after repeated attempts still produce more than 500 errors or refusals.Source: §B.5 · p. 13

B.6 Evaluation Details for Aesthetic Description

Description evaluation uses GPT to compare model responses with reference descriptions, with repeated scoring rounds used to reduce variability.

B.6 Evaluation Details for Aesthetic Description

  • GPT averages evaluations over five rounds because identical prompts can produce variable responses.The prompts assess completeness, preciseness, and relevance against a reference description.Source: §B.6 · p. 13
  • The evaluation prompts score whether descriptions cover aesthetic dimensions, avoid contradictory judgments, and stay relevant to aesthetic attributes and terminology.The listed attributes include composition, color, lighting, focus, sentiments, and improvement suggestions.Source: §B.6 · pp. 13–14
  • Table 8 reports AVA and TAD66K results using PLCC and SRCC, with colors marking the best and second-best values.The supplied passage identifies the table’s scope and metrics but does not provide its cell values.Source: Table 8 · p. 14

C.1 Result on Supervised Scoring

The supervised-scoring experiment reports that UNIAA-LLaVA surpasses EAT on AVA and TAD66K; qualitative examples also reveal limitations in camera-shot perception and differences in aesthetic descriptions.

C.1 Result on Supervised Scoring

  • UNIAA-LLaVA achieves superiority over state-of-the-art EAT on AVA and TAD66K in supervised scoring.The reported setup uses the last two vision-encoder layers, projected separately and summed before a score-prediction head; Table 8 reports PLCC/SRCC.Source: §C.1 · p. 14

C.2 Qualitative Analyse on Perception

  • Some MLLMs show inadequate spatial perception when identifying camera-shot types in UNIAA-QA examples.The responses are illustrated in Figure 6.Source: §C.2 · p. 14, Figure 6 · p. 14

C.3 Qualitative Analyse on Description

  • On high-aesthetic images, UNIAA-LLaVA gives professional descriptions, while on low-aesthetic images it can identify shortcomings that other models tend to praise.The authors note that identifying image shortcomings is more challenging than praising an image, and report that IDCP data could improve aesthetic discernment and expression.Source: §C.3 · p. 14

D.1 UNIAA-LLaVA Limitations

The limitations discussion notes reduced general-purpose performance and remaining distance from human performance, while also identifying constraints in the benchmark’s data and description-evaluation protocol.

D.1 UNIAA-LLaVA Limitations

  • UNIAA-LLaVA’s improved aesthetic abilities coincide with decreased performance on general-purpose tasks, especially heavy-reasoning and language tasks.The authors caution that use beyond aesthetic perception and understanding may produce undesired outcomes.Source: §D.1 · p. 14
  • UNIAA-LLaVA still performs worse than humans, and its converted data mostly comprises natural in-the-wild images.The authors identify natural and artistic image datasets, AI-generated content, and video aesthetics as areas for further work.Source: §D.1 · p. 17

D.2 UNIAA-Bench Limitations

  • The benchmark’s attribute and option counts retain numerical deviations despite efforts to balance them.The authors say they will continue refining the benchmark.Source: §D.2 · p. 17
  • Description evaluation remains subjective, and the five-round GPT-assisted protocol cannot eliminate hallucinations.Human ratings of MLLM descriptions also vary between individuals.Source: §D.2 · p. 17
  • UNIAA-Bench’s planned expansion includes art paintings, AI-generated content, videos, and movie clips with corresponding aesthetic dimensions.The authors also plan to explore better protocols for evaluating low-level visual descriptions.Source: §D.2 · p. 17
Loading 2404.09619v1…