Source-linked AI summary
Seeing Red, Thinking Bad: Color Bias in Vision Language Models
Kohsuke Ide, Ryousuke Yamada, Yoshihiro Fukuhara, Hirokatsu Kataoka, Yutaka Satoh
TL;DR
VLM decisions can diverge from human values, but how visual styling affects their interpretation of rendered text remains unclear. This paper introduces semantics-preserving visual prompts and finds that color and contrast changes systematically bias sentiment and VQA outputs.
Problem
The biases underlying many VLM decisions remain unclear, motivating study of how visual and textual information processing affects safe, reliable AI decision-making.
Method
The paper introduces Stealth Visual Prompts, semantics-preserving changes to text color and contrast, to systematically test their effects on VLM behavior and representations.
Results
VLMs are sensitive to rendered-text styling: color biases sentiment predictions, while reduced contrast increases salient-span reliance and decoy-driven VQA errors.
Takeaways & Limitations
Ordinary formatting can act as an implicit control channel, motivating evaluation and robustness methods that explicitly account for visual styling in deployed VLM pipelines.
Takeaways & Limitations
The study uses English-only stimuli, RGB-defined intensity levels, and an OCR proxy with limited scope.
Abstract
from arXiv · showhide
Vision language models (VLMs) are increasingly used in industrial decision-making systems, such as recruitment support and recommendation. This motivates careful analysis of how VLMs process visual and textual information. In this work, we study how VLMs interpret text rendered as an image, and investigate the influence of visual styling biases. To this end, we introduce Stealth Visual Prompts, which subtly change visual styling of text, such as color and contrast, while preserving semantic content. Using these prompts, we systematically control the visual styling of words in text and measure their impact on the analysis performed by VLMs. We further analyze how such visual perturbations affect the latent representations of the vision encoder. From our experiments, we observed that coloring positive words in green consistently shifts sentiment predictions toward a positive direction. As a result, VLMs often fail to properly account for negative words present in the text. Our analysis suggests that this behavior is correlated with changes in the latent representations of the vision encoder induced by color variations. In addition, we show that reducing text--background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. These results suggest that the visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding. Project page: https://github.com/KohsukeIde/color-bias-vlm
1 Introduction
VLMs can be influenced by low-level visual styling even when rendered text preserves semantic content, raising concerns about opaque biases in AI decision-making. The paper introduces Stealth Visual Prompts to systematically study these effects across sentiment analysis, VQA, and vision-encoder representations.
- Motivation: VLMs’ unclear training-data biases and potential misalignment with human values motivate examining how they process visual and textual information.The authors frame this analysis as necessary for safer and more reliable AI decision-making systems.
- Problem: Semantically equivalent text can yield inconsistent VLM outputs when presented as tokens versus rendered text, because attributes such as color and contrast influence predictions.These visual attributes can affect model behavior without changing the underlying words.
- Method: Stealth Visual Prompts alter text styling in perceptible ways while preserving semantic meaning, enabling controlled evaluation of visual bias.The framework includes short- and long-sentence sentiment sets and a VQA set that controls contrast on answer-irrelevant words.
- Findings: The experiments analyze how text styling distorts vision-encoder representations and changes end-to-end behavior in subjective sentiment and objective question-answering tasks.The authors characterize these latent-space shifts as a previously underexplored vulnerability relevant to VLM robustness.
2 Related Work
Prior work benchmarks VLM understanding of text-rich images and documents, while reporting cross-modal inconsistency between semantically identical inputs. This work studies visually induced biases through appearance changes that preserve lexical content, including reduced-contrast effects on hallucination.
- Text-as-image understanding and cross-modal sensitivity: Text-rich image understanding is evaluated by benchmarks including TextVQA, DocVQA, ChartQA, and TextCaps.Related benchmarks also cover scene-text VQA/OCR, infographic/document QA, and chart/figure reasoning.
- Text-as-image understanding and cross-modal sensitivity: Recent studies report cross-modal inconsistency, with semantically identical content producing different outputs across input modalities.
- Stealth Visual Prompts: Stealth Visual Prompts vary visual appearance while preserving lexical content, without adding words or explicit instructions, to analyze visually induced biases.Prior work on color reliability and shortcut associations motivates color as a semantically neutral control signal.
- Hallucination: Reduced text contrast can increase hallucinated outputs even when the affected words are task-irrelevant.This finding connects visual styling perturbations to established concerns about hallucination in VLMs.
3 Methodology
The methodology controls text-as-image styling while preserving string content, then evaluates behavioral changes across sentiment and VQA settings. It also uses diagnostic probes to relate these effects to model representations and decoy-copying behavior.
- Auxiliary diagnostic probes: Two controlled single-word probes diagnose color- and contrast-induced effects, including CLIP semantic projections onto ten bipolar axes.These probes complement the end-to-end Stealth Prompt Testset by examining representation-level mechanisms.
- Stealth Visual Prompts: Stealth Visual Prompts perturb text rendering while keeping the underlying string fixed, focusing on human-like formatting changes in color and contrast.The method treats these variations as ordinary emphasis or readability adjustments rather than explicit instructions.
- Stealth Prompt Testset: The Stealth Prompt Testset targets local lexical integration, structured discourse with positional effects, and access-limited text understanding.Its datasets include short-sentence sentiment, longer structured sentiment, and contrast-manipulated VQA conditions.
- Evaluation metrics: Sentiment outputs are restricted to POSITIVE, NEUTRAL, or NEGATIVE labels and mapped to scalar scores for measuring bias relative to an all-black baseline.The methodology defines sentiment bias for each visual condition and interprets positive or negative values as directional shifts from baseline.
- Evaluation metrics: VQA evaluation reports token-level F1 and uses Induced Error Rate to measure copying of visually salient decoy words under reduced visibility.IER is explicitly treated as a decoy-copying measure rather than a general VQA accuracy metric.
4 Experiments and Results
Experiments show that visual styling can systematically bias VLM sentiment and VQA behavior while leaving rendered text semantically unchanged. Susceptibility varies by model and setting: color effects dominate locally mixed sentiment, position can dominate structured discourse, and reduced contrast increases saliency-driven errors for some models.
- Short-sentence Sentiment Set: Qwen2-VL-7B had the largest color susceptibility, with a Total Range of 0.90, while LLaVA variants had much smaller ranges of 0.04–0.12.Qwen2-VL-7B showed up to +0.42 positive bias when positive words were green/blue and down to -0.48 when negative words were red.
- Short-sentence Sentiment Set: Coloring sentiment-bearing words produced directional shifts for Qwen2-VL-7B and IDEFICS2-8B, with stronger intensity generally amplifying the bias.LLaVA variants remained close to zero across most hues and intensities.
- Long-sentence Sentiment Set: Under structured long-sentence discourse, models often followed positional heuristics, leaving color as a residual influence that depended on discourse structure.Color cues could dominate locally mixed settings, whereas position could dominate structured layouts.
- Auxiliary probe: hue: Hue systematically shifted CLIP image embeddings along text-defined semantic axes even when the rendered word was unchanged.Valence projections were relatively higher around green (∼120◦) and dipped around blue (∼250◦); these diagnostic shifts are consistent with the color-induced sentiment biases but do not establish a universal mechanism.
- VQA Stealth Set: As non-salient context became less visible, Induced Error Rate increased from 24% to 36% for IDEFICS2-8B and from 19% to 25% for LLaVA-Vicuna-7B.LLaVA-Mistral-7B stayed relatively flat at 24–27%, while Qwen2-VL-7B remained low at 4–6%.
- OCR proxy: The OCR proxy showed a non-linear readability transition, with accuracy rising sharply over a narrow contrast range that varied by model and font size.This calibration signal is consistent with increased saliency-driven VQA errors when non-salient context becomes less accessible, but it does not directly model long-context VQA reading.
5 Discussion
The discussion shows that visual styling can steer VLM interpretation despite unchanged text: color biases sentiment through vision-encoder representation shifts, while reduced contrast increases reliance on salient spans and VQA errors. These sensitivities create reliability risks, motivating text normalization, OCR cross-checks, style-invariance evaluation, and broader testing beyond the study’s limitations.
- Visual styling sensitivities: VLMs can produce different outputs for semantically identical rendered text, with color acting as an implicit control channel for sentiment predictions.Recoloring a small subset of sentiment-bearing words biases predictions, while hue shifts move vision-encoder embeddings along interpretable axes such as valence.
- Visual styling sensitivities: Lower text–background contrast increases reliance on visually salient spans, producing more decoy-driven VQA errors.A single-word OCR proxy also shows model-dependent readability transitions, suggesting contrast changes can shift visual-text access regimes.
- Reliability and safety: Benign or adversarial styling can steer VLM decisions without changing underlying text, creating reliability and safety risks for document and UI-screenshot pipelines.Suggested safeguards include rendered-text normalization, OCR-based answer cross-checks, and style-invariance checks in evaluation suites.
- Limitations and future work: The study is limited to English stimuli, RGB-defined intensity levels, and a narrow OCR proxy.Future work should examine fonts, layout, and multilingual scripts.
6 Conclusion
The paper introduces Stealth Visual Prompts as a controlled method for studying rendered-text understanding in VLMs and shows that visual styling can bias interpretation. These findings motivate evaluation and robustness methods that explicitly account for formatting in deployed VLM pipelines.
- Methodological contribution: Stealth Visual Prompts preserve semantics while systematically testing how word-level visual styling affects rendered-text understanding in VLMs.The methodology changes the visual rendering of text without changing its semantic content.
- Key findings: Word-level color styling can bias sentiment predictions and sometimes override contradictory lexical evidence, while structured long sentences can induce positional heuristics.These effects are demonstrated by the Stealth Prompt Testset.
- Mechanistic probes: CLIP embeddings shift with hue along semantic axes, and an OCR proxy reveals model-dependent readability transitions.The auxiliary probes help contextualize the observed visual-style effects.
- Implications: Formatting is not purely cosmetic for text-as-image inputs because it can act as an implicit control channel in deployed VLM pipelines.The findings motivate evaluation protocols and robustness methods that explicitly account for visual styling.