Source-linked AI summary
Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Xiaofu Chen, Stella Frank, Yova Kementchedjhieva
TL;DR
The paper asks whether vision encoders contain conceptual information about canonical color beyond visible image color. It uses controlled grayscale inputs, object–color data, and linear probes across vision encoders and VLM representations, finding persistent canonical-color decodability and model-dependent redistribution after VLM post-training.
Problem
It remains unclear whether encoders represent canonical color as object-level conceptual knowledge or simply expose chromatic cues from the input.
Method
The study constructs an object–color dataset and probes frozen vision representations for canonical color and object identity using RGB, grayscale, counterfactual, and VLM settings.
Results
Canonical color remains linearly decodable after direct color removal, aligns more with canonical than surface color, and can reappear after the visual interface or in decoder-side states.
Takeaways & Limitations
Canonical color provides a controlled lens for tracing object-level semantic information across vision encoders and the vision–language stack.
Takeaways & Limitations
The conclusions concern canonical color and linear accessibility, not all visual attributes or all information contained in the representations.
Abstract
from arXiv · showhide
Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.
1 Introduction
The paper asks whether vision encoders represent canonical color as object-level conceptual knowledge rather than merely exposing chromatic input cues. It uses grayscale controls, linear probes, and VLM comparisons to trace this information through vision–language representations.
- Canonical color tests whether encoders encode object-level conceptual knowledge or simply expose chromatic cues available in the input.
- The study removes visible color by grayscale conversion while preserving object structure, then probes canonical color and object identity from frozen representations.
- RGB inputs provide an upper-bound setting because probes can exploit visible pixels rather than canonical object knowledge.
- The VLM analysis compares matched pre- and post-training vision towers and decoder-side visual-token representations.
- Canonical color remains linearly decodable after direct color cues are removed, while its information becomes differently accessible across VLM components after post-training.
2 Related Work
Related work treats canonical color as a semantic attribute that probing can examine, while emphasizing that RGB probes may exploit visible pixels and that probe results require controlled interpretation.
- Semantic feature norms represent object concepts through typical properties, including perceptual attributes such as color.
- Prior image-encoder probes may predict color from full-color RGB images, leaving visible pixel cues confounded with canonical color knowledge.
- Input-side controls remove chromatic information and reduce luminance cues to distinguish canonical color from directly visible color.
- Linear-probe accuracy measures accessible information but does not by itself establish that a model uses that information in final predictions.
- VLM research identifies language priors, dataset bias, hallucination, weak object–attribute binding, and Concept Association Bias as influences on multimodal predictions.
3 Data and Probing Setup
The study builds a manually screened object–color dataset, applies color-controlled image preprocessing, and uses layer-wise linear probes to compare canonical-color and object-identity accessibility.
- Canonical Color Dataset: The dataset contains 708 object classes associated with 10 canonical-color labels and targets five images per class.
- Canonical Color Dataset: Canonical-color pairs combine prior data with Wikidata entries whose single has-color value is mapped to ten basic colors and manually screened.
- Canonical Color Dataset: Images are selected from Wikidata links and Google Search results, retaining up to five clear depictions consistent with each object's canonical color.
- Input-side Controls: RGB is treated as an upper-bound condition because visible pixel color can directly support canonical-color prediction.
- Input-side Controls: Grayscale conversion removes chromatic information, and per-image histogram equalization reduces brightness cues that may correlate with color.
- Linear Probing: Frozen vision encoders are probed for 10-way color classification and 708-way object identity, with layer-wise trajectories compared rather than absolute accuracies.
- Linear Probing: Canonical color labels describe object categories, so color decodability may reflect category-level object identification or other correlated image cues.
4 Controlled Probing of Canonical Color in Vision Encoders
Controlled probing shows that canonical color remains linearly accessible from grayscale inputs across vision encoders, while its accessibility is associated with—but not fully determined by—object-class representations.
- 4.2 Validating the Color-controlled Setting: Grayscale sharply reduces color probing while preserving decodable object-class recognition, validating removal of direct color cues without eliminating object structure.Visual-CounterFact further tests whether the residual signal reflects actual surface color or canonical object color.
- 4.2 Validating the Color-controlled Setting: Actual-color probing stays near the majority baseline, whereas canonical-color probing remains above baseline on original and counterfactual images.This indicates that the residual signal aligns more closely with canonical object color than counterfactual surface color.
- 4.3 Canonical Color Decodability Across Encoders: All five encoders remain above the 13.98% majority-class baseline for canonical-color probing under grayscale inputs, with most improving from early to middle layers.The compared encoders are CLIP, SigLIP, DINOv2, ViT-MAE, and Swin-V2, using layer-wise patch-averaged representations.
- 4.3 Canonical Color Decodability Across Encoders: Canonical-color accuracy increases monotonically with DINOv2 scale, from 46.2% to 51.8% for Gray and from 41.8% to 48.0% for Gray+HE.The trend has Spearman’s ρ = 1.00 across the evaluated scales.
- 4.4 Relation to Object-Class Recognition: Object-class recognition and color probing generally improve with depth and show strong positive layer-wise correlation, but their trajectories are not always matched.Color probing can peak before object recognition or decline while object recognition continues improving, indicating partial rather than complete alignment.
5 VLM Post-training and the Location of Semantic Decodability
VLM post-training changes where canonical-color and object information is linearly accessible under grayscale inputs. In some models, decodability decreases in the standalone vision tower but reappears at the visual interface or in decoder-side representations.
- 5.1 Model-dependent changes in the vision tower: Post-training lowers linear-probe accuracy in the Molmo and mPLUG-Owl vision towers, with a larger reduction for object recognition than canonical color.These effects are model-dependent: PaliGemma largely preserves the original SigLIP pattern.
- 5.1 Model-dependent changes in the vision tower: PaliGemma’s post-VLM vision tower closely follows the original SigLIP encoder for both canonical color and object recognition.Reduced standalone-tower decodability is therefore not universal across the matched pairs.
- 5.2 Visual-interface and decoder-side probing: Decoder-side visual-token representations recover much of the lost decodability in Molmo and mPLUG-Owl, while PaliGemma remains broadly comparable to its vision-tower pattern.Molmo recovers canonical-color decodability from the first decoder layer; object recognition rises toward the original CLIP peak later.
- 5.2 Visual-interface and decoder-side probing: In mPLUG-Owl, object-recognition accuracy rises substantially at the visual abstractor output before visual tokens enter the LLaMA decoder.The first decoder layer changes the value only slightly, while later layers further refine it.
- 5.2 Visual-interface and decoder-side probing: These results are consistent with representational reformatting rather than removal, although linear probing does not establish that the post-VLM ViT contains no object information.A blank-image baseline near the majority-class baseline suggests the recovered signal depends on visual input.
- 5.3 Prompt-level behavior under color-controlled inputs: Final VLM answers are not a clean measure of visual color perception because they conflate visual evidence, object recognition, canonical-color priors, and output normalization.The prompt-level experiment uses grayscale images and evaluates visual color, canonical color, and object recognition separately.
6 Conclusion
Canonical color remains linearly decodable from grayscale images and aligns more closely with canonical than surface color. Across VLMs, post-training can shift this information from standalone vision towers to visual-interface and decoder-side representations.
- 6 Conclusion: Canonical color remains linearly decodable from grayscale images, while RGB probing is strongly influenced by visible color.Counterfactual and object probes align the residual signal more closely with canonical than surface color and show only partial alignment with object identity.
- 6 Conclusion: VLM post-training can reduce tower-level decodability while canonical-color and object information reappear after the visual interface or within decoder-side states.The pattern is consistent with representational reformatting rather than removal.
Limitations
The study’s conclusions are scoped by its focus on canonical color and its use of linear probing. These choices support controlled comparisons but do not measure all visual attributes or all information in the representations.
- Canonical color is a controlled case study, so conclusions should not be directly generalized to attributes such as material, texture, function, or affordance.Those attributes may require different controls and evaluation protocols.
- Linear probing measures only information that is linearly accessible from the chosen readout.Nonlinear probes or downstream task-specific heads might recover additional information.
- The results should therefore be interpreted as evidence about linear decodability rather than a complete measure of information in the representations.
A Additional Dataset and Model Details
The appendix provides additional details on dataset construction, label sources, evaluation splits, and model repositories used in the experiments.
- A Additional Dataset and Model Details: The appendix documents dataset construction, label sources, evaluation splits, and model repositories.
A.1 Dataset Construction and Label Source
The dataset derives canonical-color labels from structured Wikidata properties, maps them to ten basic colors, and supplements object-linked images with search results and quality filtering. Sampling retains label imbalance while capping overrepresented Pokémon concepts and prioritizing less frequent colors.
- Label source: 708 object concepts are labeled using single-value Wikidata color properties mapped to ten basic color categories.The categories are black, blue, brown, gray, green, pink, purple, red, white, and yellow.
- Image collection: Images are collected from Wikidata links and Google Search, then filtered for object identity, visual quality, and consistency with the canonical color.Manual inspection controls image inclusion but does not assign or revise labels.
- Sampling: The dataset preserves imbalance across canonical-color labels instead of equalizing category sizes.This choice retains the observed label distribution for the probing dataset.
- Sampling: Pokémon concepts are capped at 50 while prioritizing the less frequent gray, brown, and red categories.The cap addresses overrepresentation in the candidate concepts.
- Class distribution: The final label counts range from 48 gray concepts to 99 white concepts across the ten canonical-color categories.White is the most frequent label, while gray is the least frequent.
A.3 Evaluation Splits and Variable Image Counts
Color probing evaluates generalization to unseen object concepts, while object recognition uses within-class image hold-outs because class-level hold-out would remove the target label from training. Concept-level color predictions are averaged across available images, with performance compared against a 14% majority baseline.
- Color probing split: Color probing uses five-fold object-class cross-validation, assigning all images of each concept to one fold.Each test fold contains object concepts unseen during probe training, so the split evaluates generalization to new concepts rather than new images of known concepts.
- Color prediction: At test time, image-level color-probability vectors are averaged within each object concept to produce one prediction per concept.The procedure accommodates concepts with different numbers of available images.
- Color evaluation: Color results are compared with an approximately 14% majority-class baseline without rebalancing the color-label distribution.Cross-validation defines held-out concepts but does not equalize label frequencies.
- Object recognition split: Object recognition uses a within-class image hold-out because holding out an entire class would remove its target label from training.For concepts with at least two images, one image is held out; single-image concepts are training-only.
B Numerical Results and Statistical Tests
The numerical analysis reports best-layer probing accuracy and statistical tests for encoder comparisons and matched vision towers before and after VLM post-training. Grayscale canonical-color accuracy exceeds the majority baseline across five encoders, while post-training effects vary substantially across model pairs.
- Statistical tests: Best-layer accuracy is reported with 95% confidence intervals, using one-sample t-tests against the majority baseline and paired t-tests for RGB-to-grayscale differences.All results use histogram-equalized grayscale inputs and five folds.
- Vision encoders: All grayscale canonical-color accuracies are significantly above the 13.98% majority baseline, and every RGB-to-grayscale reduction is significant.These results are reported for the five vision encoders in Table 4.
- Effect magnitude: The PaliGemma object-recognition difference is 1.9 percentage points despite being consistent across folds.Statistical significance is interpreted alongside effect magnitude.
- Pre-/post-VLM vision towers: The two CLIP-based matched pairs show substantial post-training reductions, particularly for object recognition, whereas the SigLIP–PaliGemma pair changes little in absolute magnitude.Table 5 compares standalone vision towers before and after VLM post-training.