PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images

Yiming Chen, Junlin Han, Tianyi Bai, Shengbang Tong, Filippos Kokkinos, Philip Torr

arXiv:2511.22805v1cs.CVcs.LGcs.MM

TL;DR

MLLMs can identify and describe image content, but their alignment with human judgments of subjective image properties remains limited. This paper introduces CogIP-Bench and uses post-training to improve cognitive alignment; the paper reports improvements across most cognitive dimensions and transfer to image generation.

  • Problem

    There is no systematic framework to evaluate and enhance MLLMs’ alignment with human judgments of subjective image properties such as aesthetics, funniness, emotional valence, and memorability.

    Source: §1 · p. 1

  • Method

    The paper introduces CogIP-Bench and a post-training pipeline that uses benchmark data to teach an MLLM to predict human cognitive scores.

    Source: §1 · pp. 1–2

  • Results

    MLLMs show near-zero Spearman correlation with human preferences on memorability; SFT improves alignment across nearly all cognitive dimensions and metrics except memorability, and ImageReward improves by +22.8%.

    Source: §4.2 · p. 5, §4.4 · p. 6, §5.2 · p. 6

  • Takeaways & Limitations

    The learned cognitive alignment transfers to image generation, where the aligned MLLM guides synthesis toward more memorable, aesthetically pleasing, or emotionally resonant images.

    Source: §1 · p. 2

Abstract

from arXiv · show

While Multimodal Large Language Models (MLLMs) are adept at answering what is in an image-identifying objects and describing scenes-they often lack the ability to understand how an image feels to a human observer. This gap is most evident when considering subjective cognitive properties, such as what makes an image memorable, funny, aesthetically pleasing, or emotionally evocative. To systematically address this challenge, we introduce CogIP-Bench, a comprehensive benchmark for evaluating MLLMs on such image cognitive properties. Our evaluation reveals a significant gap: current models are poorly aligned with human perception of these nuanced properties. We then demonstrate that a post-training phase can effectively bridge this gap, significantly enhancing the model's alignment with human judgments. Furthermore, we show that this learned cognitive alignment is not merely predictive but also transferable to downstream creative tasks. By integrating our cognitively-aligned MLLM into an image generation pipeline, we can guide the synthesis process to produce images that better embody desired traits, such as being more memorable or visually appealing. Our work provides a benchmark to measure this human-like perception, a post-training pipeline to enhance it, and a demonstration that this alignment unlocks more human-centric AI.

1. Introduction

The paper addresses the lack of systematic evaluation of MLLMs’ alignment with human judgments of subjective image properties. It introduces CogIP-Bench and shows that post-training can improve this alignment and transfer to image generation.

1. Introduction

  • CogIP-Bench provides a systematic benchmark of MLLM-human cognitive alignment across aesthetics, funniness, emotional valence, and memorability.The paper presents it as a comprehensive benchmark for these four image properties.Source: §1 · p. 1
  • All evaluated MLLMs show near-zero human correlation for memorability, while average Spearman correlation on the other cognitive dimensions remains below 0.5.The introduction identifies memorability as a particularly challenging subjective property for current models.Source: §1 · p. 1
  • Post-training on CogIP-Bench data improves an MLLM’s prediction of human cognitive scores, and the aligned model can guide Qwen-Image synthesis toward desired traits.The paper names memorability, aesthetic appeal, and emotional resonance as target traits.Source: §1 · pp. 1–2
  • Figure 1 presents benchmark examples, post-training results across three MLLMs, and image-generation results after swapping the MLLM backbone.Source: Figure 1 · p. 2

2. Related Work

Existing MLLM benchmarks assess broad visual and textual capabilities, while research on image cognition has often focused on other model types or individual properties. The paper positions CogIP-Bench as a systematic benchmark for subjective visual cognition across four dimensions.

2.1. MLLMs and Their Evaluations

  • Existing all-round MLLM benchmarks assess capabilities such as visual grounding, perception, OCR, and compositional reasoning, but not alignment on subjective image properties.Source: §2.1 · p. 2

2.2. Human Cognition in AI

  • Earlier AI cognition research examined human-model decision alignment [17] [64] and neural-network mechanisms [51] [52], while image studies addressed properties such as memorability [18].Source: §2.2 · pp. 2–3
  • Prior MLLM cognition work focused on spatial reasoning and causal inference, while studies of image cognition generally covered single dimensions or vision-only models.CogIP-Bench is presented as the first systematic benchmark covering four subjective image-cognition dimensions.Source: §2.2 · p. 3

3. CogIP-Bench

CogIP-Bench evaluates whether MLLMs’ image-score predictions align with human preferences across four cognitive dimensions. Its construction draws on dimension-specific datasets and score sources, with balanced sampling and a defined training and test set for each property.

3. CogIP-Bench

  • Each benchmark datapoint combines an image, a cognition-specific prompt, and a human-preference score as ground truth.Source: §3 · p. 3

3.1. Benchmark Construction

  • The four dimensions are aesthetics, funniness, emotional valence, and memorability, with aesthetic scores drawn from LAION-Aesthetic-Predictor-V1 and humor scores from HumorDB.The aesthetic data use LaMem images; HumorDB contains varied image types with humor scores collected through three crowdsourced tasks.Source: §3.1 · p. 3
  • Table 1 reports MSE and MAE for each dimension and includes dimension-specific vision-only models as references.Its colors mark best and worst entries separately for open-source and API-based models.Source: Table 1 · p. 4
  • Emotional-valence data come from FindingEmo, while memorability data and scores come from LaMem.Emotional valence distinguishes positive or negative quality from arousal; LaMem’s scores derive from large-scale human memory experiments.Source: §3.1 · p. 4
  • The benchmark balances sampled datapoints across score buckets, using five ranges for all dimensions except emotional valence, which uses three.Memorability scores are rescaled, and emotional-valence scores are scaled for MSE and MAE calculations.Source: §3.1 · p. 4

3.2. Benchmark Details

  • Each cognitive property has 800 training datapoints and 120 test datapoints, each comprising an image, a cognition-specific prompt, and a numeric ground-truth score.The prompts are curated to explain the relevant cognitive property; Figure 2 shows examples.Source: §3.2 · p. 4

4. Benchmarking Results

The section evaluates MLLMs’ alignment with human judgments of image cognition, then tests supervised fine-tuning (SFT) and its effects on other benchmarks. SFT improves alignment across nearly all cognitive dimensions, while memorability remains difficult.

4. Benchmarking Results

  • The section presents CogIP-Bench results, describes the post-training pipeline, and examines both alignment changes and performance on other benchmarks.Source: §4 · p. 4

4.1. Experimental Settings

  • Dimension-specific comparisons use the LAION aesthetic predictor, CLIP-based regressors for funniness and emotional valence, and HumanMem for memorability.Source: §4.1 · p. 5

4.2. Benchmark Results on MLLMs

  • Baseline results show near-zero human correlation for memorability; open-source models align more strongly on aesthetics and emotional valence, while API models do better on funniness and emotional valence.The section also reports that better performance on other benchmarks is associated with relatively strong CogIP-Bench performance, and GPT-5 performs better than other API models.Source: §4.2 · p. 5

4.3. Post-training Method

  • The post-training approach uses supervised fine-tuning, classifying image judgments before mapping them to scores and using soft labels to preserve numerical proximity between digit tokens.Standard cross-entropy loses numerical distance information; the soft-label distribution for numeric targets is mixed with a triangular function.Source: §4.3 · p. 5

4.4. Post-training Results

  • SFT improves alignment across nearly all cognitive dimensions and metrics, except memorability, where all models remain difficult to predict.Table 2 reports normalized MSE and MAE for each cognitive subtask before and after SFT.Source: §4.4 · p. 6, Table 2 · p. 6
  • On other benchmarks, Gemma3-12B-it demonstrates substantial improvements across all categories, while Qwen2.5-VL-7B and Llama-3.2-11B maintain comparable performance with acceptable variance.The benchmark sections are Vision-Centric, OCR, General, and Knowledge.Source: §4.4 · p. 6, Table 3 · p. 6

5. Image Generation Results

The section compares Qwen-Image generations using the base and SFT-aligned MLLM backbones, including prompts designed to elicit cognitive traits. Across automatic metrics and a user study, the aligned backbone improves image scores and is preferred more often.

5. Image Generation Results

  • The generation comparison uses Qwen-Image with a consistent random seed, comparing its base MLLM backbone with the SFT variant that shares that backbone.Source: §5 · p. 6

5.1. Prompt Preparation

  • ChatGPT-5 generated 100 text-to-image prompts for each of five sets: Aesthetics, Funniness, Emotional, Memorability, and General.Source: §5.1 · p. 6

5.2. Image Metrics Results

  • ImageReward improves by +22.8%, and all reported general image-quality and human-preference metrics improve with the SFT-aligned backbone.Source: §5.2 · p. 6, Table 4 · p. 7
  • Emotional cognition scores gain +19%, the largest increase, while predicted cognition scores improve across all dimensions after post-training.Source: §5.2 · p. 7, Table 5 · p. 7

5.3. User Study

  • In a user study with 30 image pairs per cognitive dimension and five volunteers, the aligned model was preferred 1.7 times more often on average.Source: §5.3 · p. 7

6. Discussions

The ablations test loss, prompting, and which model component to fine-tune; the RL experiment compares GRPO with standard SFT. The findings identify gains in cognitive alignment alongside a trade-off in broader multimodal reasoning.

6.1. Ablation Study

  • Freezing the vision encoder gives the weakest cognitive alignment; freezing the LLM performs slightly better, especially for aesthetics and funniness, but reduces broader multimodal reasoning.Source: §6.1 · p. 8, Table 6 · p. 8
  • The proposed prompting strategy yields the largest cognitive-alignment gains across dimensions, while soft-label loss generally improves alignment.Source: §6.1 · p. 8, Table 7 · p. 8

6.2. Cognition-Alignment with RL

  • GRPO improves aesthetics and emotional cognition alignment over standard SFT, but degrades broader multimodal reasoning tasks.Source: §6.2 · p. 8, Table 8 · p. 8

7. Conclusion

The conclusion argues that MLLMs can acquire some human-like image-cognitive alignment and that this alignment can transfer to creative applications.

  • The work concludes that subjective cognitive visual understanding is within reach for MLLMs.Source: §7 · p. 8
  • The learned cognitive alignment is presented as transferable to more intuitive, human-centric creative applications.Source: §7 · p. 8
  • The authors frame the work as a step toward AI systems that can both see and feel the world as humans do.Source: §7 · p. 8

A. Detailes of Results on Other Benchmarks

This appendix section reports results for three MLLMs across 12 benchmarks in four task categories, before and after SFT on CogIP-Bench.

  • The detailed results cover three MLLMs evaluated before and after SFT on CogIP-Bench across 12 benchmarks and four task categories.Source: A. Detailes of Results on Other… · p. 13
  • The table groups benchmark averages into Vision-Centric, OCR, General, and Knowledge categories.Source: Table S1 · p. 13
  • Green and red entries mark improvements and regressions, respectively, in a model's average result for a task category.Source: Table S1 · p. 13

B. Training Details

The training details identify the hyperparameters, including soft-label parameters, for three model variants. The user study compares generated image pairs using blinded volunteer preferences.

B. Training Details

  • The training hyperparameters, including soft-label parameters, are specified for Qwen2.5-VL-7B, Gemma3-12B-it, and Llama-3.2-11B-VI.The specification is summarized in Table S2.Source: Table S2 · p. 13, B. Training Details · p. 13

C. Example fo User Study

  • For each cognitive dimension, five volunteers chose between 30 randomly sampled image pairs or selected “Hard to Tell,” without knowing which model produced them.The pairs compared images generated using default or SFT-aligned MLLM backbones.Source: C. Example fo User Study · p. 13
  • Each image pair in the user study was generated from the same prompt with Qwen-Image using a consistent seed.Figure S1 demonstrates the study setup.Source: Figure S1 · p. 14

D. Examples of SFT Dataset

The SFT dataset examples span four cognitive dimensions, with figures showing samples across those dimensions. The accompanying prompts cover dataset instruction and response rewriting, as well as text-to-image prompt generation.

D. Examples of SFT Dataset

E. Prompts

  • Figure S4 shows rewriting of original simple dataset instructions, Figure S5 covers rewriting GPT ground-truth responses, and Figure S6 gives attribute-specific and general-purpose text-to-image prompts.The instruction-rewriting example uses aesthetics.Source: E. Prompts · p. 16, Figure S4 · p. 16, Figure S5 · p. 17

Related papers

Loading 2511.22805v1…