Source-linked AI summary
On Color Alignment in VAE Latent Spaces and Its Applications
Julian D. Santamaria, Kai Wang, Jesús Malo, Javier Vazquez-Corral, Alexandra Gómez-Villa
TL;DR
Color is highly structured in natural images, but how text-to-image VAEs represent it has remained little explored. The paper analyzes encoder responses and steers latents to identify shared brightness and opponent-color directions, then uses them for generation control, with ColorTuning achieving state-of-the-art accuracy on CSS3/X11 GenColorBench.
Problem
How color is represented in text-to-image VAE latents has only recently begun to be explored, despite color’s structured organization in natural images.
Source: §1 · p. 2Method
The paper characterizes latent color directions through a linear approximation of the encoder and targeted latent steering.
Source: §1 · p. 2, §3.2 · p. 5Results
81.20% vs. 51.96%: ColorTuning achieves state-of-the-art accuracy on the CSS3/X11 system of GenColorBench.
Source: §1 · p. 2Takeaways & Limitations
The same color basis supports saturation control and color transfer to match a reference palette.
Source: §5 · p. 10Large steering magnitudes can affect image content because the extracted color channels are not fully disentangled from spatial and structural features.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Variational autoencoders (VAEs) are a key part of modern text-to-image models, which generate images within their latent space. VAEs are known to disentangle the main factors of variation in the data, and color is known to be one of the most structured of these in natural images: decorrelating it yields one luminance axis and two opponent-color axes. Color should therefore be expected to emerge as a distinct factor in the VAE latent space. Yet how these latent spaces represent color remains largely unexplored. In this work, we show that the VAEs of text-to-image models share a color subspace aligned with brightness and opponent-colors. Through a linear approximation of the encoder and targeted latent steering, we find this subspace consistently across a broad range of VAEs, from SD1.5 to FLUX.2 and Z-Image. Building on this characterization, we propose three applications: ColorTuning, which achieves state-of-the-art in precise numerical color generation on the fine-grained CSS3/X11 system of GenColorBench, saturation control, to adjust the global chromatic intensity, and color transfer, to change the palette to match a reference. The code and models are publicly available at https://julian075.github.io/Color_Subspace/
1 INTRODUCTION
Natural-image color has a structured luminance and opponent-color organization, yet how color is represented in text-to-image VAE latents has only recently begun to be explored. This work identifies shared color directions in those latents and uses them for color control during generation.
1 INTRODUCTION
- Earlier unsupervised and convolutional vision systems learn achromatic and opponent-color features, with recent evidence of similar structure in vision foundation models (Gomez-Villa et al., 2025).Source: §1 · p. 2
- 81.20% vs. 51.96%: ColorTuning achieves state-of-the-art numerical color-generation accuracy on CSS3/X11 GenColorBench.Source: §1 · p. 2
- Across modern text-to-image VAEs, color subspaces align with brightness and nonlinear opponent-color axes across model families and latent sizes from 4 to 32 channels.Source: §1 · p. 2
- Linear encoder approximation and targeted latent steering characterize the basis, while its color channels remain partially entangled with image structure.Source: §1 · p. 2
- The same basis supports saturation control of global chromatic intensity and reference-driven color transfer.Source: §1 · p. 3
2 RELATED WORK
Prior work connects latent disentanglement to data structure, probes color representations in vision models, and develops several forms of color control. The section also introduces the VAE and latent-generation formulations used in the paper, including intermediate clean-latent estimates for measuring target-object color.
2 RELATED WORK
- Disentanglement research has relied almost exclusively on synthetic benchmarks; natural-image color provides a test grounded in luminance and opponent axes (Buchsbaum & Gottschalk, 1983; Ruderman et al., 1998).Source: §2 · p. 3
- Earlier text-to-image VAE color probes used uniformly colored patches, whereas this work analyzes natural images across five VAEs with 4 to 32 latent channels.Source: §2 · p. 3
- Prior precise-color methods alter text conditioning, cross-attention, or latents; the closest HSL recoloring approach is FLUX-specific, unlike ColorTuning’s recovered VAE basis and measured CIELAB target shifts.Source: §2 · p. 3
2.1.1 VARIATIONAL AUTOENCODERS
- The VAE encoder maps images to a probabilistic latent and its decoder models image likelihood; the β-weighted ELBO balances reconstruction against KL regularization, with β > 1 strengthening regularization.Source: §2.1.1 · p. 3
- Text-to-image VAEs use β ≪ 1 with perceptual and adversarial losses, remain frozen after training, and encode images as spatial latents with C channels.Source: §2.1.1 · p. 3
2.1.2 LATENT DIFFUSION AND FLOW MATCHING
- Diffusion corrupts clean VAE latents and predicts noise, whereas flow matching uses a linear data-noise path and predicts velocity; intermediate clean-latent estimates can be decoded to measure target-object color during generation.Source: §2.1.2 · pp. 3–4
3 COLOR IN AUTOENCODER LATENT SPACE
Latent-channel traversals reveal color changes across autoencoders, while a linear approximation of the encoder shows opponent-color directions among the latent space’s structure. These color directions coexist with spatial information and are not generally isolated in individual channels.
3.1 LATENT CHANNELS ENCODE COLOR
- Traversing a latent channel by adding a constant offset at every spatial position progressively shifts decoded-image color across autoencoders and pretrained text-to-image VAEs.The result is observed in models with 3, 4, 8, and 32 channels, without a learned probe.Source: §3.1 · p. 4, Figure 2 · p. 5
- Color edits from channel traversals are limited: larger offsets eventually alter spatial content as well as color.As |δ| increases, ∆E00 grows while grayscale MS-SSIM stays near 1 for small offsets and then drops.Source: §3.1 · p. 4
- Individual channels generally mix colors or produce similar color directions, but three-channel VAEs and AEs align closely with brightness and two opponent-color axes.The paper notes that this AE alignment is not predicted by the VAE explanation of axis alignment.Source: §3.1 · p. 4
- When color is distributed across correlated channels, PCA of their color responses recovers one luminance and two opponent-color directions.The paper uses this mechanism in its applications.Source: §3.1 · p. 5
3.2 LINEAR APPROXIMATION OF THE VAE
- The linear approximation treats an image as a deviation from average gray and represents the autoencoder response with a first-order matrix approximation.The paper reports that the encoder and decoder approximations explain a large fraction of signal energy and have small errors relative to the original signal.Source: §3.2 · pp. 5–6
- Across text-to-image VAEs, two principal opponent chromatic directions emerge in the encoder’s singular vectors, aligning their latent spaces with opponent-color channels.The analysis uses a linear approximation of the encoder and finds this pattern across all evaluated models.Source: §3.2 · p. 6
4 APPLICATIONS
The paper recovers an explicit color basis for each VAE and uses it to steer text-to-image generation for precise color control, saturation adjustment, and reference-based color transfer. ColorTuning leads on CSS3/X11, while the same basis also supports qualitative control of saturation and palettes.
4.1 CHARACTERIZING THE COLOR BASIS OF A VAE
- The color basis is recovered by perturbing latent channels for seven reference objects, stacking measured color-change sensitivities, and applying SVD to isolate three directions.The directions target luminance and two opponent-color axes; generation-time perturbations are localized to the object and decay during the final steps.Source: §4.1 · pp. 6–7
4.2.1 STEERING THE GENERATOR
- ColorTuning steers an object toward its prompt’s exact RGB or HEX color by measuring its intermediate color and predicting a displacement along the recovered basis.The displacement predictor is trained from initial and target colors paired with empirically observed shifts; inference uses a proxy color name and applies steering within the object mask.Source: §4.2 · p. 7, §4.2.1 · pp. 7–8
4.2.2 RESULTS OF COLORTUNING
- On full GenColorBench, ColorTuning is best on every backbone for CSS3/X11, reaching 81.20% on FLUX; NumColor remains ahead on ISCC-L3.Table 1 reports numerical color accuracy on the mini subset and full benchmark, with latent-channel counts in parentheses.Source: Table 1 · p. 7, §4.2.2 · pp. 8–9
- On GenColorBench’s mini subset, ColorTuning reaches about 81% on CSS3/X11 for FLUX.2 and Z-Image and more than doubles ColorWave on SDXL, with perceptual preferences in a 22-observer study.The reported ColorWave comparison is 63.97% versus 28.05% on SDXL; ColorTuning is preferred over NumColor on FLUX and ColorWave on SDXL.Source: §4.2.2 · p. 9
- Qualitative examples show distinct hues, subtle within-family color variations, and pastel tones, with requested colors faithfully reproduced even in the challenging pastel cases.The examples cover the full spectrum as well as fine-grained differences between target colors.Source: Figure 5 · p. 8, §4.2.2 · p. 9
4.3 FURTHER COLOR APPLICATIONS DURING GENERATION
- The same basis continuously modulates image saturation, with the illustrated control spanning 100% to 50% in 10% intervals.The authors suggest this could help synthesize images within narrower color gamuts and potentially reduce the need for post-processing gamut mapping.Source: §4.3 · p. 9
- For color transfer, the method uses a reference image’s color distribution while generating the content specified by the text prompt.The authors describe this application qualitatively.Source: §4.3 · p. 9
5 CONCLUSIONS
Color structure in autoencoder latents aligns with achromatic and opponent-color patterns, and an explicit basis supports color control during generation. The approach has a boundary: large steering magnitudes can affect image structure as well as color.
5 CONCLUSIONS
- Recovering an explicit color basis turns latent color structure into a tool for color control during generation, including numerical color generation, saturation control, and color transfer.ColorTuning achieves state-of-the-art accuracy on the fine-grained CSS3/X11 system of GenColorBench; saturation control and transfer are demonstrated qualitatively.Source: §5 · pp. 9–10
- The color structure appears in AEs, VAEs, and VQ-VAEs, and a linear encoder approximation links its leading singular vectors to achromatic and opponent-color patterns.Traversing a single latent channel shifts the decoded image’s color.Source: §5 · p. 9
- Large steering magnitudes can distort image structure because extracted color channels are not fully disentangled from spatial and structural features.The distortion occurs when the predicted displacement from source to target color is large.Source: §5 · p. 10
A.1 LLM USAGE STATEMENT
The statement reports that LLMs assisted with manuscript editing and code development.
A.1 LLM USAGE STATEMENT
- LLMs were used for grammar correction and to rewrite and refine parts of the manuscript.The statement also reports LLM assistance with code development.Source: §A.1 · p. 14
A.2.1 REPRESENTATION LEARNING AND DISENTANGLEMENT IN VAES
VAE disentanglement depends on the relationship between data variance and generative factors, and its identifiability requires assumptions. Natural-image color offers a test of whether VAEs capture structured factors beyond synthetic benchmarks.
A.2.1 REPRESENTATION LEARNING AND DISENTANGLEMENT IN VAES
- What a VAE disentangles is dictated by the principal directions of its training data, and disentanglement occurs only when data variance matches generative factors.Prior work links this alignment to the diagonal posterior and local principal directions of the data.Source: §A.2.1 · p. 14
- Identifiability results require additional assumptions, while VAE disentanglement has been evaluated almost exclusively on synthetic benchmarks.Color provides a natural-image test: its statistics decorrelate into luminance and opponent-color axes, and this work examines those axes in VAE latents.Source: §A.2.1 · p. 14
A.2.2 COLOR REPRESENTATIONS IN VISION FOUNDATION MODELS
Prior studies probe color representations in vision models, including generative-model VAEs, but often use flat stimuli or examine few models. This work instead studies natural images across five VAEs.
A.2.2 COLOR REPRESENTATIONS IN VISION FOUNDATION MODELS
- Studies of vision encoders report limited alignment with human color judgments, while Stable Diffusion patch probes recover intensity and opponent hue axes.A separate FLUX study identifies an HSL-like subspace and uses it to steer generated-object color.Source: §A.2.2 · p. 14
- Uniformly colored images reveal where colors map in latent space, but not how color is encoded alongside real images’ spatial content.The cited generative-model studies examine only one or two VAEs, leaving cross-model consistency open.Source: §A.2.2 · p. 14
- The paper probes natural images across five VAEs with 4 to 32 latent channels to examine color representation beyond flat stimuli.Source: §A.2.2 · p. 14
A.2.3 PRECISE COLOR CONTROL IN TEXT-BASED IMAGE MODELS
Precise color-control methods intervene through text conditioning or inference-time changes. ColorTuning instead recovers a color basis for pretrained VAEs and learns target-color shifts from measured CIELAB outcomes.
A.2.3 PRECISE COLOR CONTROL IN TEXT-BASED IMAGE MODELS
- Text-conditioning methods learn color tokens or ground numerical codes with a CIELAB-aligned ColorBook, while inference methods alter cross-attention or optimize latents.Source: §A.2.3 · pp. 14–15
- Pach et al. (2026) recolor objects using an HSL mapping from eight anchor colors, but their interpretation is specific to FLUX and flat color patches.Source: §A.2.3 · p. 15
- ColorTuning recovers luminance and opponent-color directions in any pretrained VAE, then learns target displacements from measured CIELAB outcomes.Source: §A.2.3 · p. 15
A.3 LATENT CHANNEL SHIFTS
The section evaluates latent-channel color shifts across autoencoder families and models, then assesses linear approximations and color-control applications. In a user study, ColorTuning is preferred over both tested competitors.
A.3 LATENT CHANNEL SHIFTS
- Channel-shift interventions probe grayscale images in custom AE, KL-VAE, and VQ-VAE baselines, then extend to pretrained latent-diffusion and foundation-model autoencoders.Custom models were trained on Tiny ImageNet-200; foundation models include SD1.5, SDXL, SD3, FLUX, and FLUX2.Source: §A.3.1 · p. 15, §A.3.2 · p. 15, §A.3.3 · p. 15
A.4 SIMILAR COLOR SUBSPACES EMERGE IN DIFFERENT DATASETS.
- Singular-vector results from 105 natural images at 64 × 64 show opponent-color subspaces across image sets and resolutions.Source: §A.4 · p. 15
A.5 FAITHFULNESS OF THE LINEAR APPROXIMATIONS.
- About 65% of nonlinear behavior is explained by the linear approximations, whose reconstruction errors are about 12% of original-signal energy and similar to the VAEs’ errors.These results support using the singular vectors to describe sensitivity to texture and color directions.Source: §A.5 · pp. 15, 20
A.7 PIPELINES; A.7.1 TEMPORAL INJECTION SCHEDULES
- Color-basis perturbations are applied within a spatial mask over an active generation window, with temporal schedules chosen per model.The schedules include triangular and constant profiles as well as an ascending ramp.Source: §A.7.1 · p. 20, §A.7 · p. 20
A.7.2 SHIFT PREDICTOR NETWORK ARCHITECTURE
- A residual-connection MLP maps initial and target CIELAB features, including chroma and hue representations, to three-dimensional color-shift magnitudes.Source: §A.7.2 · pp. 20, 23
A.8 SATURATION CONTROL AND COLOR TRANSFER
- Saturation control scales chroma while preserving lightness and hue, using the predictor to shift colors within the latent-space gamut.Source: §A.8 · pp. 24–25
- Color transfer assigns reference palette colors to segmented scene regions and predicts a separate color-basis displacement for each region.Reference colors may come from a discrete palette or an exemplar image.Source: §A.8 · p. 25
A.9 USER STUDY
- In Thurstone Case V comparison z-scores, ColorTuning showed statistically significant preference over NumColor (2.62 ± 0.30) and ColorWave (1.67 ± 0.30), with positive scores indicating preference for ColorTuning.Source: §A.9 · p. 28