PaperlayerView exact revision on arXiv

Source-linked AI summary

The Photographer Eye: Teaching Multimodal Large Language Models to Understand Image Aesthetics like Photographers

Daiqing Qi, Handong Zhao, Jing Shi, Simon Jenni, Yifei Fan, Franck Dernoncourt, Scott Cohen, Sheng Li

arXiv:2509.18582v2cs.CV

TL;DR

Aesthetic visual understanding requires more than identifying image objects, but existing MLLMs and datasets have limitations in handling the expertise and diversity needed for detailed aesthetic analysis. The paper introduces PhotoCritique, PhotoEye, and PhotoBench, and reports advantages over existing models on existing benchmarks and PhotoBench.

  • Problem

    MLLMs can handle general visual tasks but still struggle with aesthetic understanding, while existing aesthetic datasets have limitations in scale, expertise, and diversity.

    Source: §1 · p. 2

  • Method

    The paper builds PhotoCritique from photographer and enthusiast discussions, proposes PhotoEye with language-guided multi-view vision fusion, and introduces PhotoBench covering 284 photography sub-topics.

    Source: §1 · p. 2

  • Results

    PhotoEye outperforms open-source models on existing benchmarks and baselines on PhotoBench, and is competitive with and slightly surpasses GPT-4o on existing benchmarks.

    Source: §6.1 · p. 7

  • Takeaways & Limitations

    The experiments support the paper's conclusion that PhotoEye advances aesthetic visual perception in MLLMs.

    Source: §7 · p. 9

Abstract

from arXiv · show

While editing directly from life, photographers have found it too difficult to see simultaneously both the blue and the sky. Photographer and curator, Szarkowski insightfully revealed one of the notable gaps between general and aesthetic visual understanding: while the former focuses on identifying the factual element in an image (sky), the latter transcends such object identification, viewing it instead as an aesthetic component--a pure color block (blue). Such fundamental distinctions between general (detection, localization, etc.) and aesthetic (color, lighting, composition, etc.) visual understanding present a significant challenge for Multimodal Large Language Models (MLLMs). Although some recent works have made initial explorations, they are often limited to general and basic aesthetic commonsense. As a result, they frequently fall short in real-world scenarios (Fig. 1), which require extensive expertise--including photographic techniques, photo pre/post-processing knowledge, and more, to provide a detailed analysis and description. To fundamentally enhance the aesthetics understanding of MLLMs, we first introduce a novel dataset, PhotoCritique, derived from extensive discussions among professional photographers and enthusiasts, and characterized by the large scale, expertise, and diversity. Then, to better learn visual aesthetics from PhotoCritique, we furthur propose a novel model, PhotoEye, featuring a languageguided multi-view vision fusion mechanism to understand image aesthetics from multiple perspectives. Finally, we present a novel benchmark, PhotoBench, a comprehensive and professional benchmark for aesthetic visual understanding. On existing benchmarks and PhotoBench, our model demonstrates clear advantages over existing models.

1. Introduction

Aesthetic visual understanding requires interpreting color, lighting, and composition rather than only identifying factual image elements, yet existing MLLMs and datasets have limitations. The paper introduces PhotoCritique, PhotoEye, and PhotoBench to address these gaps.

1. Introduction

  • Aesthetic understanding differs from object identification by requiring perception of chromatic properties and compositional principles, which MLLMs still struggle with.The paper identifies dataset limitations and models that overlook aesthetic-specific visual features as two contributing factors.Source: §1 · p. 2
  • PhotoCritique contains over 450K images with detailed aesthetic descriptions drawn from discussions among professional photographers and enthusiasts, rather than a small fixed annotator group.The dataset is described as around 20 times the size of existing datasets [8] [30].Source: §1 · p. 2
  • PhotoEye uses language-guided multi-view vision fusion to learn aesthetic knowledge from PhotoCritique and answer aesthetics-related questions from multiple perspectives.Source: §1 · p. 2
  • PhotoBench evaluates aesthetic understanding through questions from professional and enthusiast discussions, covering 284 photography sub-topics.The paper also reports a comprehensive study comparing PhotoEye with existing MLLMs.Source: §1 · p. 2

2. Related Works

Prior work extends MLLMs to image aesthetics, but interpretability and the scale, diversity, and professional curation of aesthetic descriptions remain concerns. Related visual-fusion approaches also leave opportunities for language-guided feature extraction and fusion.

Understanding Image Aesthetics with MLLM

  • Traditional Image Aesthetics Assessment [32] predicts aesthetic-quality scores through regression, but the paper notes its lack of interpretability.Recent MLLM approaches [8] [9] [10] [29] [30] [31] [38] explore interpretable aesthetic understanding but often have limited data scale or lack diverse, professionally curated descriptions.Source: §2 · p. 2
  • Existing aesthetic datasets [8] [30] [38] often have limited scale or lack diverse, professionally curated descriptions, while aesthetic understanding remains underexplored among MLLMs.Source: §2 · p. 2

Improving Visual Perception in MLLM

  • SPHINX [14] concatenates encoder features, MoVA [41] uses CLIP-based fusion, and Cambrian averages visual features with queries independent of language instructions.The paper proposes PhotoEye's language-guided multi-view vision fusion in response to these design limitations.Source: §2 · p. 3

3. Learning with Photography Enthusiasts and Professionals

The section motivates PhotoCritique through limitations in existing aesthetic training data, then describes mining and organizing photographer discussions into a large, diverse dataset.

3. Learning with Photography Enthusiasts and Professionals

  • Existing datasets have about 20K images and typically 57K–88K detailed descriptions, with limited genre coverage and reliance on small, fixed annotator groups.PhotoCritique instead draws on insights from hundreds of thousands of photographers and enthusiasts.Source: §3 · p. 3

3.1. Mining from Digital Photography Platforms

  • The mining pipeline uses DPC images and comments, summarizes and integrates the comments into one critique without the image, then filters out critiques lacking aesthetic information.A smaller LLM performs the filtering for efficiency.Source: §3.1 · p. 3
  • LLMs turn critiques into diverse aesthetic conversations about lighting, composition, color, emotion, and photographic techniques, then generate five critique-grounded MCQs.Prompt-based filtering rejects less informative conversation pairs.Source: §3.1 · pp. 3–4

3.2. PhotoCritique

  • PhotoCritique contains 450K aesthetic descriptions, 1.9M conversations, and 250K VQA samples across more than 70 photographic categories.The data comes from shared images and discussions by photography enthusiasts and professionals on online forums.Source: §3.2 · p. 4
  • PhotoCritique improves across all Table 1 metrics, and its average aesthetic-comment length is notably greater than that of existing datasets.The comparison includes source images, annotations, instruction-tuning pairs, and average comment length.Source: §3.2 · pp. 4–5, Table 1 · p. 4

4. Improving Visual Aesthetics Perception

Existing MLLMs often miss aesthetic features because their vision encoders are less sensitive to them than to general visual content. PhotoEye combines vision encoders through language-guided queries and feature fusion to analyze aesthetics from different perspectives.

4.1. See like Photographers

  • CLIP is less effective at discriminating aesthetics-related features, which are more densely clustered in its visual representations.Figure 8 compares aesthetics-related feature discriminability for PhotoEye and CLIP in existing works.Source: §4.1 · p. 5, Figure 8 · p. 5

4.2. PhotoEye

  • PhotoEye fuses visual features from multiple encoders, conditioning feature extraction and fusion on both the image and the aesthetics-related instruction.The resulting feature is passed through the backbone LLM to answer aesthetics-related questions.Source: §4.2 · pp. 5–6
  • A language-guided query generator uses the instruction embedding to select a query from learnable queries, tailoring feature extraction to the aesthetic concept.The paper uses a pretrained BERT text encoder and avoids the CLIP text encoder to prevent bias toward CLIP visual features in later gating.Source: §4.2 · p. 5
  • Before fusion, features from different vision encoders are interpolated to a common shape and processed with a learnable query.The query-based extraction is part of each fusion block.Source: §4.2 · pp. 5–6
  • The multimodal gating network predicts feature weights from the language embedding and visual features, then combines the features into one representation.The fused representation is further processed by a transformer block.Source: §4.2 · p. 6

5. Benchmarking Aesthetic Visual Understanding with Professionals

PhotoBench draws on photography-community reviews rather than annotator-provided annotations to form professional, diverse questions. Its generation pipeline filters for image dependence and aesthetics relevance, and its questions address practical photography knowledge.

5. Benchmarking Aesthetic Visual Understanding with Professionals

  • PhotoBench draws on Reddit PhotoCritique reviews rather than annotator-provided annotations to create expert-level questions across a wider range of scenarios.The paper identifies limited expertise and diversity among small groups of annotators as a limitation of existing benchmarks.Source: §5 · p. 6

5.1. From Insights to Questions

  • Question generation summarizes noisy comments into a critique, generates five critique-grounded multiple-choice questions per image, then filters the candidates.From 25,000 generated questions, image-free answering removed about 70%; the authors then selected the top 1,500 by average aesthetics relevance, visual dependency, and expertise scores.Source: §5.1 · p. 6

5.2. PhotoBench

  • Compared with existing works, PhotoBench is described as more diverse and expert, while their questions are often relatively simple.The authors attribute this simplicity to potentially limited annotator expertise and diversity.Source: §5.2 · p. 6
  • PhotoBench asks about practical photographic topics, including camera settings, post-processing, and photographic techniques, rather than focusing mainly on factual elements.Examples discuss reducing exposure to make a rainbow stand out and raising the shadow curve to retain texture or create a film-like aesthetic.Source: §5.2 · pp. 6–7

6. Experiments

Experiments compare PhotoEye with existing models on established benchmarks and PhotoBench, then examine the contributions and behavior of its vision fusion and learned aesthetic representations.

6.1. Results

  • On Q-Bench, PhotoEye competes despite PhotoCritique not being designed for low-level vision, and slightly surpasses GPT-4o while outperforming open-source models.The open-source comparisons include general-purpose models and models fine-tuned on low-level vision and aesthetics datasets.Source: §6.1 · p. 7, Table 2 · p. 8
  • Ablations show that both the vision fusor and PhotoCritique contribute to PhotoEye.The paper points to additional ablations and qualitative results in the appendix.Source: §6.1 · p. 7, Table 4 · p. 8
  • PhotoEye notably outperforms baselines on PhotoBench, whose professional and diverse questions demand stronger aesthetic understanding.The comparisons include UNIAA, AesExpert, and Q-Instruct; PhotoBench categories may overlap.Source: §6.1 · p. 7, Table 3 · p. 8

6.2. See through PhotoEye

  • Encoder weights indicate aesthetic features are extracted early and general features late, with CLIP and DINO dominant in the last layer.The analysis averages normalized weights over 100 samples each from compositional aesthetics, general aesthetics, and general visual tasks.Source: §6.2 · p. 8, Figure 11 · p. 8
  • CoDETR, pretrained with detection objectives, is dominant for compositional problems where object localization or arrangement is critical.The paper says Tab. 5 validates this finding by activating one encoder at a time for photographic composition.Source: §6.2 · p. 8, Table 5 · p. 8
  • PhotoEye learns discriminative aesthetics-related visual representations, which the paper describes as foundational to high-quality aesthetic responses.The paper points to a feature-level investigation in Appendix B.Source: §6.2 · p. 8

7. Conclusion

The paper presents PhotoCritique, PhotoEye, and PhotoBench for aesthetic visual understanding, and reports that PhotoEye has a clear advantage over existing models in extensive experiments.

7. Conclusion

  • The paper introduces PhotoCritique, PhotoEye with language-guided multi-view vision fusion, and PhotoBench, and reports PhotoEye’s clear advantage over existing models in extensive experiments.The conclusion frames these contributions as advancing aesthetic visual perception in MLLMs.Source: §7 · p. 9

Related papers

Loading 2509.18582v2…