PaperlayerView exact revision on arXiv

Source-linked AI summary

ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness

Yijun Liang, Ming Li, Chenrui Fan, Ziyue Li, Dang Nguyen, Kwesi Cobbina, Shweta Bhardwaj, Jiuhai Chen, Fuxiao Liu, Tianyi Zhou

arXiv:2504.10514v3cs.CVcs.AIcs.CLcs.LG

TL;DR

Existing VLM benchmarks often give limited attention to color-centric reasoning and robustness. The paper introduces COLORBENCH and evaluates 32 VLMs, finding weak scaling, benefits from reasoning steps, and cases where color cues mislead models.

  • Problem

    Existing VLM benchmarks often overlook color-centric reasoning and robustness, leaving unclear how well VLMs understand color and withstand color perturbations.

    Source: §1 · p. 2

  • Method

    The paper introduces COLORBENCH, a dedicated 11-task benchmark across color perception, reasoning, and robustness, and evaluates 32 VLMs.

    Source: §1 · pp. 2–3

  • Results

    Scaling persists but is weak and mainly depends on language-model components; reasoning steps improve accuracy and robustness, while color can mislead models on illusion and mimicry tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings emphasize the need for more sophisticated model architectures that integrate deeper color reasoning capabilities.

    Source: §4 · p. 9

    Vision-encoder scaling conclusions are constrained by mainstream VLMs' narrow encoder choices, with most using 300–400M encoders.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Color plays an important role in human perception and usually provides critical clues in visual reasoning. However, it is unclear whether and how vision-language models (VLMs) can perceive, understand, and leverage color as humans. This paper introduces ColorBench, an innovative benchmark meticulously crafted to assess the capabilities of VLMs in color understanding, including color perception, reasoning, and robustness. By curating a suite of diverse test scenarios, with grounding in real applications, ColorBench evaluates how these models perceive colors, infer meanings from color-based cues, and maintain consistent performance under varying color transformations. Through an extensive evaluation of 32 VLMs with varying language models and vision encoders, our paper reveals some undiscovered findings: (i) The scaling law (larger models are better) still holds on ColorBench, while the language model plays a more important role than the vision encoder. (ii) However, the performance gaps across models are relatively small, indicating that color understanding has been largely neglected by existing VLMs. (iii) CoT reasoning improves color understanding accuracies and robustness, though they are vision-centric tasks. (iv) Color clues are indeed leveraged by VLMs on ColorBench but they can also mislead models in some tasks. These findings highlight the critical limitations of current VLMs and underscore the need to enhance color comprehension. Our ColorBenchcan serve as a foundational tool for advancing the study of human-level color understanding of multimodal AI.

1 Introduction

ColorBench addresses limited evaluation of color-dependent VLM abilities by testing color perception, reasoning, and robustness. Its evaluation finds weak scaling, benefits from reasoning steps, and cases where color cues mislead models.

1 Introduction

  • Existing VLM benchmarks often underemphasize color-centric reasoning and robustness, leaving color understanding insufficiently assessed.Color also provides clues for tasks beyond color naming, including object detection, scene interpretation, planning, and applications such as medical care and remote sensing.Source: §1 · pp. 1–2
  • COLORBENCH evaluates color perception, reasoning, and robustness through 11 fine-grained tasks grounded in real-world applications.Source: §1 · p. 2
  • The paper introduces a dedicated benchmark and reports a fine-grained evaluation of 32 VLMs to expose limitations in color understanding.Source: §1 · pp. 2–3
  • Scaling persists but is weaker and mainly depends on language-model components; model performance gaps are relatively small.The reported relationship between vision-encoder size and performance is not significant, with current encoder choices limited.Source: §1 · p. 3
  • Adding reasoning steps improves ColorBench performance, including on color robustness tasks.The paper notes that color robustness had not been investigated by the community.Source: §1 · p. 3
  • VLMs leverage color clues in most tasks, but color can mislead them on illusion and mimicry tasks; grayscale conversion can improve accuracy.Source: §1 · p. 3

2 COLORBENCH Construction

COLORBENCH evaluates VLM color understanding across perception, reasoning, and robustness, with 1,448 instances and 5,814 image-text questions across 11 tasks. Explicit color constraints focus the perception and reasoning questions on color-related abilities.

2 COLORBENCH Construction

  • COLORBENCH contains 1,448 instances and 5,814 image-text questions across 11 tasks in Color Perception, Color Reasoning, and Color Robustness.Source: §2 · p. 3

2.1 Taxonomy

  • The taxonomy organizes color understanding into three dimensions and 11 detailed axes, motivated by existing criteria and real-world applications.Source: §2.1 · p. 3

2.1.1 Color Perception

  • Color Perception covers detecting and interpreting colors through Color Recognition, Color Extraction, and Object Recognition.Source: §2.1.1 · pp. 3–4

2.1.2 Color Reasoning

  • Color Reasoning tests drawing conclusions from color understanding and prior knowledge across seven tasks, including proportion, counting, illusions, mimicry, and color blindness.Source: §2.1.2 · p. 4

2.1.3 Color Robustness

  • Color Robustness measures whether predictions remain correct when the same color-independent question is paired with recolored versions of an image.Source: §2.1.3 · p. 4

2.2 Data Curation

  • Perception and Reasoning images are mainly collected manually from online sources, while Color Extraction, Color Blindness, and Color Illusion images are programmatically generated.Source: §2.2 · p. 4
  • For robustness, 493 seed images from CV-Bench receive nine variants using three region strategies and hue shifts of 90°, 180°, or 270°; experts filter unnatural or negligible edits.Source: Figure 3 · p. 4, §2.2 · p. 5

2.3 Evaluation Metrics

  • Perception and Reasoning use task- and category-level accuracy; robustness requires a correct seed-image answer and correct answers for every recolored variant, then aggregates across seeds.Source: §2.3 · p. 5

3 Experimental Results

COLORBENCH results show that VLMs remain limited in color understanding, with modest gaps between models, while larger models and reasoning steps often improve performance. Color cues help on many tasks but can mislead models on illusion and mimicry.

3.1 Main Results

  • Humans achieve the highest performance on every evaluated task, while GPT-4o and Gemini-2-flash perform best among the VLMs overall.Color Extraction is excluded from the human evaluation because humans are not attuned to precise color-code differences.Source: §3.1 · p. 5, Table 1 · p. 6
  • Most models exceed 60% on Color Recognition, but proprietary models reach only middle-tier Color Extraction performance and do not lead Object Recognition.The authors suggest Object Recognition may align strongly with common training recipes containing abundant object-detection images.Source: §3.1 · pp. 5–6
  • Color Proportion peaks at 58.0% and Color Counting at 43.1%, while LLaVA-OV-72B is the only model above 50% on Object Counting.Most Color Illusion scores fall between 30% and 50%; GPT-4o without CoT is the only model above 50%, and most open models score below 30% on Color Blindness.Source: §3.1 · p. 6
  • Only four models exceed 80% on Color Robustness; GPT-4o scores 46.2% and Gemini-2 70.7%, with CoT gains of +23.7% and +2.9%, respectively.The four models above 80% use relatively large InternViT-6B vision encoders.Source: §3.1 · p. 7

3.2 Further Findings

  • Performance correlates with model size, especially language-model size, but evidence for a vision-encoder-size correlation is insufficient given limited encoder choices.The reported correlations between overall model size and P&R Overall and Robustness are 0.7619 and 0.7390, respectively.Source: §3.2 · p. 7, Table 2 · p. 7, Figure 4 · p. 7
  • GPT-4o and Gemini-2 reach 53.9% P&R Overall accuracy, only +2.0% above the best open-source model; the best group-1 model reaches 41.5%.The best group-1 result is 10.4% below the best open-source model, and proprietary models can show weaker robustness than a 7B model.Source: §3.2 · p. 8, Table 3 · p. 8
  • CoT improves average P&R Overall accuracy by +4.35% across GPT-4o and Gemini-2, benefiting Color Extraction and most Color Reasoning tasks but harming Color Illusion.CoT also consistently improves Color Robustness; the authors suggest that it may raise confidence in correct answers after reasoning.Source: §3.2 · pp. 8–9, Table 4 · p. 8
  • Grayscale conversion reduces accuracy on most tasks, but often improves it on Color Illusion and Color Mimicry, where color can mislead models.Color Counting and Color Blindness show mixed effects, with roughly half of model accuracies increasing and half decreasing.Source: §3.2 · p. 9, Figure 5 · p. 9

4 Conclusion, Limitation, and Future Works

The paper presents COLORBENCH as a comprehensive benchmark of VLM color understanding and reports evaluation of 32 models. Its manually collected annotations support consistency but limit dataset scale, style diversity, and category coverage.

4 Conclusion, Limitation, and Future Works

  • COLORBENCH is introduced as a benchmark for VLM color perception, reasoning, and robustness, evaluated across 32 widely used models.The authors say their findings emphasize the need for more sophisticated architectures integrating deeper color reasoning.Source: §4 · p. 9
  • Manual data collection, annotation, and assessment support consistency but inevitably constrain dataset scale, style diversity, and category coverage.Future work aims to automate collection and expand to diverse tasks involving color with texture, shape, and spatial relationships.Source: §4 · p. 9
  • The authors identify analysis of different visual encoders and language models as a way to further investigate how VLMs process color information.Source: §4 · p. 9

A.1 VLM Benchmarks

Existing VLM benchmarks cover many capabilities, but the cited work describes no unified evaluation of visual perception, reasoning, and robustness alongside deeper color understanding. Prior color evaluations address separate aspects rather than a holistic assessment.

A.1 VLM Benchmarks

  • Existing benchmarks assess text-centric, vision-centric, and specialized capabilities, but none cited provide an integrated evaluation of visual perception, reasoning, and robustness.Existing color-related questions in some benchmarks mainly address basic perception and recognition rather than deeper color reasoning and robustness.Source: §A.1 · p. 14

A.2 Color Evaluation

  • Prior color studies examine textual color terms, fine-grained discrimination, color foils, compositional color, or illusions, but the authors identify no holistic assessment framework.COLORBENCH is presented as a comprehensive, quantitative, automated benchmark for color-related VLM abilities.Source: §A.2 · p. 14

B Data Sources

COLORBENCH draws data from websites, public benchmarks, and generated images. Its data-source and recoloring-strategy details are referenced in tables.

B Data Sources

C Detailed Generation Process for Robustness

The appendix details how COLORBENCH’s robustness cases are generated and how its tasks and evaluations are organized. It also describes the model, prompting, and human-evaluation setups.

C Detailed Generation Process for Robustness

  • Robustness cases alter color in the target segment, largest irrelevant segment, or entire image, while preserving shapes and context.Hue shifts of 90°, 180°, and 270° produce nine variations per seed image; experts filter the results, leaving 493 seed images and color-invariant questions.Source: C Detailed Generation Process for… · p. 15

D COLORBENCH Categories and Questions

E Implementation Details

  • The evaluation spans 32 VLMs, includes CoT prompting for GPT-4o and Gemini-2-flash, and evaluates GPT-o3 on perception and reasoning tasks.Open-source models below 70B parameters use one NVIDIA A100 80GB GPU; larger models use four.Source: E Implementation Details · p. 19

F Evaluation Prompts

  • The standard prompt requests only the answer letter; the CoT prompt instead asks models to think step by step before answering.Both prompts provide an image, instruction, and answer options.Source: F Evaluation Prompts · p. 19

G Human Evaluation

  • Human evaluation uses three participants, each completing 50 samples per category on selected perception and reasoning tasks.Color Extraction is excluded because humans are generally not sensitive to fine-grained color-code differences.Source: G Human Evaluation · p. 19

H Reasoning Models with Thinking Process

Human evaluators lead on most tasks, while GPT-o3 scores highest on Object Recognition, Color Proportion, and Color Comparison. GPT-o3’s advantage is especially large on Color Proportion.

H Reasoning Models with Thinking Process

  • GPT-o3 leads humans and other VLMs by more than 12% on Color Proportion, while humans score highest on most tasks.GPT-o3 also leads on Object Recognition and Color Comparison, where its margins over humans are each under 3%.Source: H Reasoning Models with Thinking Process · p. 19, Table 8 · p. 20
  • GPT-o3 benefits from analytical tools for precise image assessment and continuous exhaustive visual search when estimating proportions.The authors connect these capabilities to its stronger Color Proportion results.Source: H Reasoning Models with Thinking Process · p. 20
  • GPT-o3 outperforms both CoT-prompted models on remaining tasks except Color Blindness, where it trails GPT-4o (CoT) by 3.7%.The authors suggest its longer reasoning path may lead to overthinking on this task, which requires precise color recognition and holistic spatial perception.Source: H Reasoning Models with Thinking Process · p. 20
  • Substantial performance gaps from human evaluators persist for VLMs, particularly on reasoning-intensive tasks.Source: H Reasoning Models with Thinking Process · p. 20

I Qualitative Analysis of Failure Cases

Failure analyses find that VLMs can attend to relevant regions yet still misinterpret color or produce incorrect answers. Modality comparisons indicate that color cues matter especially for Color Recognition and Object Recognition.

I Qualitative Analysis of Failure Cases

  • In most analyzed Color Recognition and Object Recognition failures, both models attend to relevant regions but still predict incorrectly, indicating color-interpretation errors rather than grounding deficiencies.The analysis compares Qwen2.5-VL-3B failures with 7B successes; attention maps are shown for color and object recognition tasks.Source: I Qualitative Analysis of Failure Cases · p. 20, Figure 17 · p. 21, Figure 18 · p. 21
  • On whole-image reasoning tasks, the 3B and 7B models show similar attention patterns despite differing answers; on targeted tasks, both attend correctly while the 3B model often fails.The authors infer that the reasoning-component difference mainly explains divergence on whole-image tasks, while both models remain weak in color interpretability.Source: I Qualitative Analysis of Failure Cases · p. 20, Figure 19 · p. 21, Figure 20 · p. 22

J Effect of Different Modalities

K Fine-tuning Experiments on ColorBench

Fine-tuning Qwen2.5-VL models with LoRA updates to the LLM component improves overall accuracy and can benefit some ancillary reasoning tasks. However, transfer is not uniformly positive.

K Fine-tuning Experiments on ColorBench

L.1 VLM Size & Model Performance for Each Task

Across tasks, larger VLMs tend to perform better, while the available vision-encoder sizes make their scaling effect difficult to assess. Within model families, more parameters generally correspond to better performance on most tasks.

L.1 VLM Size & Model Performance for Each Task

  • Larger VLMs show higher performance across the Perception and Reasoning tasks as model size grows.Figure 26 to 35 present task-wise correlations between log-scaled VLM parameter sizes and performance.Source: §L.1 · p. 25, Figure 26 · p. 26

L.2 Vision Size & Model Performance for Each Task

  • Vision-encoder scaling is difficult to assess because most VLMs use one encoder type, with sizes generally between 300-400M.Figure 36 to 40 show performance against log-scaled vision-encoder size.Source: §L.2 · p. 27, Figure 36 · p. 27

L.3 Performance for Each Model Family on Each Task

  • Within model families, models with more parameters tend to perform better on most tasks.Figures 41 to 47 compare task performance across models within the same families.Source: §L.3 · p. 28, Figure 41 · p. 28

M Samples Cases

The examples illustrate that reasoning steps often improve performance, while color cues help on most tasks but can mislead on others. Cases also show coupled vision-language failures and examples of both majority-correct and majority-incorrect answers.

M.1 Effect of CoT

  • Adding reasoning steps can improve performance on most tasks, but gains are limited for recognition and performance worsens on Color Illusion.Recognition tasks rely heavily on accurate vision-encoder cognition; on Color Illusion, reasoning focuses more on misleading surroundings and foreground relationships.Source: §M.1 · pp. 30, 34, Figure 59 · p. 34, Figure 61 · p. 34

M.2 Effect of Grayscale

M.3 Failure with LLM and Vision

  • A representative failure combines a vision error identifying pink and purple flowers instead of red and yellow with incorrect language-model reasoning.The language model reasons from the wrong color categories and arrives at an erroneous answer.Source: §M.3 · p. 36, Figure 72 · p. 36

M.4 Easy Cases

  • The paper presents sample cases where a majority of VLMs answer correctly.Examples span Color Recognition, Object Recognition, Color Comparison, Color Mimicry, Color Extraction, Color Proportion, Object Counting, and Color Robustness.Source: §M.4 · p. 37, Figure 73 · p. 37, Figure 80 · p. 38

Color Robustness

  • In the Color Robustness examples, most VLMs keep answers unchanged in one case but change them over image color variations in another.Figures 80 and 91 show these contrasting cases.Source: Figure 80 · p. 38, Figure 91 · p. 40

M.5 Difficult Cases

  • The paper presents sample cases where a majority of VLMs answer incorrectly.Examples include Color Recognition, Object Recognition, Color Comparison, Object Counting, Color Extraction, Color Proportion, Color Counting, Color Illusion, and Color Mimicry.Source: §M.5 · p. 39, Figure 81 · p. 39, Figure 88 · p. 39
Loading 2504.10514v3…