PaperlayerView exact revision on arXiv

Source-linked AI summary

Paint-Anything: Unified Any-Color Control for Image Generation and Editing

Ji Xie, Dewei Zhou, Xinyu Huang, Zhennan Chen, Xun Wang

arXiv:2609.20816v2cs.CVcs.AIcs.LG

TL;DR

Professional design calls for specifying object colors with precise hex values, while prior approaches often use specialized representations or inference procedures. Paint-Anything learns a shared hex-prompt interface from object-level supervision; it improves color-fidelity scores on both generation and editing benchmarks and leads the compared methods on average CompColor score.

  • Problem

    Prior color-control approaches often rely on dedicated representations or specialized inference procedures, while generation, colorization, and editing are often treated as separate problems.

    Source: §1 · p. 1

  • Method

    Paint-Anything learns one prompt-native hex interface for generation and editing from object-level supervision, combining real-image labels with pure-color anchors used only at high-noise timesteps.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    85.3% for ACBench-T2I and 28.3% for ACBench-Edit: Paint-Anything’s color-fidelity scores improve over the FLUX.2-4B base model, and it achieves the highest average CompColor score among compared methods.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The reported findings support a shared prompt-native hex interface for object-level color control across generation and editing.

    Source: §1 · p. 2, §6 · p. 10

    The current training data do not include palette-specific supervision.

    Source: §5 · p. 10

Abstract

from arXiv · show

Professional design requires any-color control: the ability to specify an object's target color with any 24-bit hex value for image generation and editing. Prior work has explored color generation, editing, and colorization, but often relies on dedicated color representations or specialized inference procedures. Advances in large language models offer a simpler starting point: even compact models can associate hex values with color semantics. We present Paint-Anything, which learns a shared hex-prompt interface for generation and editing through object-level color supervision. We develop a data pipeline that constructs Paint-500K from real images through object grounding, perceptual color labeling, and editing-pair synthesis. Since shadows make real-image labels only approximate colors, we complement this supervision with pure-color anchors whose pixels exactly match their paired hex values. These anchors are used only at high-noise timesteps, leaving low-noise training to natural images. We further introduce Any Color Benchmark (ACBench), comprising ACBench-T2I and ACBench-Edit, to measure object-level hex color fidelity across both tasks. On FLUX.2-4B, Paint-Anything improves ACBench-T2I and ACBench-Edit scores by 85.3% and 28.3%, respectively, relative to the base model, with ablations supporting the training recipe. It also achieves the highest average CompColor score among the compared methods.

1 Introduction

Professional design calls for specifying an object’s color with a precise 24-bit hex value, but prior color-control approaches often use specialized representations or inference procedures. Paint-Anything uses object-level hex supervision to bring generation and editing under one prompt-native interface.

1 Introduction

  • Prior color-control methods often rely on task-specific modules, learned color tokens, or training-free inference techniques, and can be difficult to scale, transfer, or use across separate color tasks.Professional design needs explicit hex values rather than broad color names such as “blue” or “red.”Source: §1 · p. 1
  • Paint-Anything keeps explicit hex values in the text prompt, using one interface for any-color generation and editing.A 4B language model’s plausible interpretations of raw hex strings motivate using them as a text interface.Source: §1 · p. 2, Figure 2 · p. 2
  • Paint-500K provides object-level hex-labeled supervision from real images, while high-noise-only pure-color anchors supply exact color references because shadows make real-image labels approximate.The paper presents timestep-gated anchors as strengthening prompt-native color control.Source: §1 · pp. 2–3
  • 85.3% for generation and 28.3% for editing: Paint-Anything improves ACBench color-fidelity scores over the FLUX.2-4B base model and achieves the highest average CompColor score among compared methods.ACBench measures object-level color fidelity on ACBench-T2I and ACBench-Edit.Source: §1 · pp. 2–3
  • ACBench evaluates object-level hex color fidelity for generated and edited object regions through ACBench-T2I and ACBench-Edit.The paper reports gains across backbones and independent benchmarks.Source: §1 · pp. 2–3

2 Related Work

Prior color-control work includes learned color representations and inference-time techniques for generation, alongside specialized approaches to editing and colorization. Paint-Anything instead uses object-level hex supervision and a shared prompt interface across generation and editing.

2.1 Color control in text-to-image generation

  • Color-control methods for text-to-image generation learn or modify color representations or use inference-time guidance, optimization, or attention/value manipulation.These approaches can rely on extra control pathways, costly inference, or task-specific assumptions that limit unified task coverage.Source: §2.1 · p. 3
  • NumColor learns numerical color embeddings, while BBQ-to-Image uses bounding boxes and RGB triplets; Paint-Anything uses hex codes in object descriptions and editing instructions without a dedicated color encoder or box-layout representation.The shared hex-prompt approach is described as unifying generation and editing.Source: §2.1 · p. 3

2.2 Color editing and image colorization

  • Color-editing and colorization methods cover language-guided or region-aware editing, palette and histogram recoloring, image-reference conditioning, and masked or semantic colorization.Paint-Anything focuses on numerical color values and the same hex-conditioned prompt interface for generation and editing.Source: §2.2 · pp. 3–4

3 Method

Paint-Anything uses one flow-matching denoising model and a shared hex-prompt interface for generation and editing. Its training combines object-level real-image supervision with high-noise pure-color anchors and data pipelines for color labeling and editing-pair synthesis.

3.1 Preliminaries

  • Generation and editing share a flow-matching denoising objective, differing only in whether the clean source-image latent is concatenated with the noisy target latent.The model predicts velocity from the noisy target latent, timestep, text tokens, and optional image-condition tokens.Source: §3.1 · p. 4

3.2 Training pipeline

  • Explicit <color> tags mark 24-bit hex values, and joint training uses the same prompt format for generation and editing.Editing samples additionally include a source image encoded as a clean condition-latent branch; empirical results report improved object-level color fidelity and compositional binding from tag wrapping.Source: §3.2 · p. 5
  • Pure-color anchors pair sampled 24-bit RGB values with solid-color images, providing clean hex-to-RGB supervision alongside approximate real-image color labels.Anchors train only at high-noise timesteps, while real-image generation and editing samples use the full timestep range; high-noise gating improves both tasks over ungated anchors.Source: §3.2 · p. 5

3.3 Data pipeline

  • MeanShift clustering in CIELAB adapts cluster count to each object's color distribution and reduces redundant shading-related splits compared with fixed-K k-means.The pipeline grounds caption-relevant objects, masks them, filters clusters by pixel coverage, and assigns the largest retained cluster's hex label to the corresponding noun phrase.Source: §3.3 · pp. 5–6, Figure 5 · p. 5
  • Paint-500K includes 400K generation samples and 100K editing samples, with multi-object captions supporting compositional binding and single-object crops emphasizing direct color perception.Editing sources are recolored single-object images, while original photographs serve as targets; filtering checks scene preservation, meaningful recoloring, and instruction consistency.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate object-level hex-color fidelity in generation and editing, alongside compositional color binding. Paint-Anything outperforms its base checkpoint and evaluated specialized systems, with ablations supporting the training recipe.

4.1 Benchmarks and metrics

  • ACBench includes 1,000 T2I prompts and 500 edit prompts; CompColor’s hex protocol substitutes canonical hex values while retaining object compositions.ACBench covers single-object, two-object, and edit settings; the T2I splits contain 500 prompts each.Source: §4.1 · pp. 6–7

4.2 Experimental setup

  • Evaluation compares open-source and color-specialized baselines; NumColor is represented by its published NCU results because its code and checkpoints were unavailable.CompColor is evaluated with named-color and hex prompts.Source: §4.2 · p. 8

4.3 Quantitative results

  • A finetuned 8B model exceeds the 56B model by 16.88 points on ACBench-T2I and 6.70 points on ACBench-Edit.Off-the-shelf ACBench Overall and Edit scores increase across models with 8B, 17B, and 56B total parameters.Source: §4.3 · p. 9
  • On CompColor, finetuning raises the hex-prompt average from 0.38 to above the base model’s named-color average, while the named-color average rises from 0.72 to 0.79.The base model’s hex-prompt average is 0.38, versus 0.72 for its named-color average.Source: §4.3 · p. 9
  • Paint-Anything beats the strongest specialized baseline by 22.04 points on ACBench-T2I and 15.19 on ACBench-Edit; it also improves compositional binding over its base checkpoint.The main-results table covers ACBench scores and CompColor, whose scores use a separate scale.Source: Table 1 · p. 7, §4.3 · pp. 8–9

4.4 Ablation study

  • Full finetuning exceeds rank-256 LoRA, while color-token wrapping improves bare-hex scores; ungated anchors raise generation but lower editing.The complete recipe is best across ACBench generation, editing, and CompColor.Source: §4.4 · p. 9, Table 2 · p. 9
  • High-noise gating beats ungated anchors by 4.75 T2I points and 5.18 Edit points; thresholds 0.7–0.9 all improve both tasks over ungated anchors.The selected gate threshold is tgate = 0.8.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 Qualitative results

  • Paint-Anything better matches requested colors across object categories and color families; Figures 7–8 show generation across hex values and source-image recoloring.Figure 7 uses the same random seed for the requested-color examples.Source: Figure 7 · p. 8, Figure 8 · p. 8, §4.5 · p. 10

5 Limitations and Future Work

The current training data do not include palette-specific supervision; future work could add it and extend the unified interface to more color-control tasks.

5 Limitations and Future Work

  • Current training data omit palette-specific supervision, which the authors identify as a direction for future work.They also propose extending the unified interface to a broader range of color-control tasks.Source: §5 · p. 10

6 Conclusion

Paint-Anything learns a unified, prompt-native any-color capability for generation and editing. It improves ACBench scores over the base model and earns the highest average scores among compared methods on CompColor and GenColorBench NCU.

6 Conclusion

  • Paint-Anything presents any-color control as a unified prompt-native capability for generation and editing, with object-level hex supervision supporting color binding and numerical color understanding.Paint-500K supplies this supervision through perceptual color clustering.Source: §6 · p. 10
  • 85.3% for generation and 28.3% for editing are Paint-Anything’s ACBench color-fidelity gains over the base model.It also achieves the highest average scores among compared methods on CompColor and GenColorBench NCU.Source: §6 · p. 10
  • Timestep-gated pure-color anchors add 4.75 points on ACBench-T2I and 5.18 points on ACBench-Edit over ungated anchors.The conclusion also reports that empirical analysis motivates MeanShift over fixed-K clustering for Paint-500K color labeling.Source: §6 · p. 10

Supplementary Material Overview

The supplementary material outlines additional benchmark, evaluation, training, and human-preference details.

Supplementary Material Overview

  • Supplementary sections describe how ACBench relates to existing benchmarks and provide a GenColorBench NCU transfer check.They also cover ACBench construction, prompt sampling, and baseline settings.Source: Supplementary Material Overview · p. 15
  • The supplement documents Paint-500K composition, the training mixture, data-building tools, and prompt templates for grounding, filtering, and recaptioning.It also includes additional training-data examples.Source: Supplementary Material Overview · p. 15
  • Further material covers pure-color generation trajectories, anchor-resolution ablations, editing by source-to-target color distance, evaluator settings, and human-preference results.It also reports ACBench reliability analyses and a training/evaluation separation check.Source: Supplementary Material Overview · p. 15

A Relation to Existing Color Benchmarks

ACBench complements existing color benchmarks with explicit 24-bit hex-to-object fidelity evaluation for generation and editing. Its construction and independent evaluation protocols support comparison across these settings.

A Relation to Existing Color Benchmarks

  • ACBench targets explicit 24-bit hex-to-object fidelity across generation and editing, complementing benchmarks for compositional binding, numerical colors, and implicit color concepts.GenColorBench NCU remains a single-object T2I numerical-color test; ACBench also covers two-object binding and image editing.Source: A Relation to Existing Color Benchmarks · p. 15
  • The GenColorBench NCU check follows its official protocol, using GroundingDINO+SAM2 localization, OneHue color extraction, and CIEDE2000 scoring with color-neighborhood tolerance.Table 4 compares averages across the same three categories; quoted NumColor scores were not rerun.Source: A Relation to Existing Color Benchmarks · p. 15, Table 4 · p. 15
  • Mean NCU rises from 34.00 to 57.89 on the same FLUX.2-4B backbone, exceeding the published FLUX + NumColor result of 51.90.The 23.89-point gain over the base and 5.99-point margin over FLUX + NumColor accompany improvements in all three categories.Source: Table 4 · p. 15, A Relation to Existing Color Benchmarks · p. 15

B ACBench Construction and Evaluation

  • ACBench-T2I has 1,000 prompts, split between 500 single-object and 500 two-object cases; ACBench-Edit has 500 real-image recoloring prompts.Each editing prompt specifies one target object and its requested hex color.Source: B ACBench Construction and Evaluation · p. 16
  • The 500 editing sources come from a high-quality real-image dataset separate from Paint-500K, and one manually inspected mask fixes the evaluated region across methods.Evaluation prompts are independently constructed from Paint-500K captions and use different formats.Source: B ACBench Construction and Evaluation · p. 16
  • ACBench samples targets from the 24-bit RGB space and scores object color with an MAE-to-score mapping; independent checks use released CompColor and official GenColorBench protocols.The ACBench protocol localizes the requested object and uses SAM3 with sRGB-MAE.Source: B ACBench Construction and Evaluation · p. 16

B.1 Baseline Settings and Result Sources

The reproduced baselines use official configurations except for a matched FLUX.2-4B editing CFG of 2.0. Results combine evaluations and quoted scores from different sources, with provenance specified per benchmark and model.

B.1 Baseline Settings and Result Sources

  • Reproduced baselines follow their official open-source and Hugging Face configurations, except FLUX.2-4B editing uses CFG 2.0.The editing guidance value is matched across comparisons.Source: §B.1 · p. 16
  • CompColor scores for SD1.5 and FLUX.1 in Table 1 are quoted from ColorBind, while their ACBench scores come from separate evaluations.Additional quoted CompColor baselines appear in Appendix C.Source: §B.1 · p. 16
  • NumColor NCU results are means of its three published category scores, not ACBench reruns; cross-model results compare complete systems.Same-backbone finetuning and ablation comparisons instead assess the training recipe.Source: §B.1 · p. 16

C CompColor Hex Protocol

The hex-translated CompColor protocol preserves the benchmark’s object compositions and splits while replacing named colors with canonical hex codes. Its results concern per-object color binding, with the three-object extension evaluated as a target-object diagnostic rather than strict joint correctness.

C CompColor Hex Protocol

  • CompColor tests per-object color binding with fixed two-object prompt structure, reducing linguistic variation while varying color-object assignments.Its failure modes include color leakage, swapped assignments, and shared dominant hues.Source: C CompColor Hex Protocol · p. 16
  • Close and Distant splits group color pairs by CIELAB perceptual distance, testing fine discrimination between similar colors versus assignment under high contrast.Single subsets check one-color, one-object rendering in isolation.Source: C CompColor Hex Protocol · pp. 16–17
  • CompColor scoring combines object localization and a color-similarity check, with higher normalized 0-to-1 scores indicating better object-level binding.The hex protocol changes only color tokens; object tokens and pair structure remain fixed.Source: C CompColor Hex Protocol · p. 17
  • The hex protocol replaces color words with canonical 24-bit values while preserving object compositions and Single, Close, and Distant splits.This removes named-color cues while retaining any numerical-color knowledge in the pretrained text encoder.Source: C CompColor Hex Protocol · p. 17
  • The main comparison aligns with released two-object baselines; the three-object extension is a complementary diagnostic, and additional original named-color baselines are reported separately.Those additional rows are copied from ColorBind and are not re-evaluated under hex prompts; the paper does not test whether each could adapt to hex inputs.Source: C CompColor Hex Protocol · pp. 17–18, Table 5 · p. 17, Table 6 · p. 18
  • Paint-Anything remains ahead under both protocols, while the three-object evaluation scores only the designated target object.Thus, the extension tests color control in scenes with three colored objects, not strict three-way joint correctness.Source: C CompColor Hex Protocol · p. 17

D Training Data and Mixture

Paint-500K combines real-image generation and editing examples with pure-color anchors, using object-level color labels and a timestep gate for the anchors. The appendix also details prompt construction and the object-mask-based ACBench evaluator.

D Training Data and Mixture

  • Paint-500K contains 500K samples: 400K T2I generation examples and 100K editing examples, mixed in training with auxiliary pure-color grounding.The T2I stream has 100K single-object and 300K multi-object samples; the 10K solid-color anchors are sampled only at t ∈ [0.8, 1.0].Source: D Training Data and Mixture · p. 18
  • Object-color labels come from grounded boxes segmented by SAM3, with CIELAB MeanShift clusters retained under pixel-coverage thresholds and the largest cluster used for a single dominant-color label.Retained clusters must each cover at least 15% of object pixels and jointly cover at least 90%, or the instance is discarded.Source: D Training Data and Mixture · p. 18
  • In a 100-sample spot check, 96 dominant-color labels agreed with the human-perceived dominant object color.Real-image labels remain approximate because shadows can produce multiple plausible RGB values; pure-color anchors pair hex codes with solid-color targets.Source: D Training Data and Mixture · p. 18

E Pure-Color Generation Trajectories

F Prompt Templates and Data-Generation Tools

  • The data-generation prompts direct grounding and caption refinement to identify separable objects, preserve exact hex strings, and insert color tags alongside object descriptions.Edit instructions are refined against source and edited images, preserving intent and mentioning visible changes only when supported by the images.Source: F Prompt Templates and Data-Generation… · pp. 19–21

G Evaluator Settings

  • ACBench uses SAM3 masks and masked-pixel mean RGB, then computes channel-wise MAE against the target hex; failed target localization receives zero score.For editing, the source-object mask is reused on the edited image without dilation.Source: G Evaluator Settings · p. 21

H ACBench Reliability

Reliability checks find that ACBench’s model ordering is stable across alternative color estimators and localization settings, while supplementary analyses examine anchor resolution, large editing shifts, human preferences, data separation, and training examples.

H ACBench Reliability

  • ACBench’s ordering persists across alternative region-color estimators, SAM3 thresholds, and localization-failure handling.For editing, Ours scores 0.7557 versus Base 0.5890 under Mean RGB, and all alternatives preserve the order with Spearman ρ=1.0; T2I Ours−Base gaps range from +0.3156 to +0.3455 across estimators. In the threshold sweep, Paint-Anything stays above Base with margins of +0.2482 to +0.3308 across the full grid; excluding failures raises its score from 68.58 to 70.93 without changing ranking.Source: Table 7 · p. 21, H ACBench Reliability · p. 22, Table 8 · p. 22

I Pure-Color Anchor Resolution

  • With the rest of training fixed, 256 × 256 anchors cost 5.48 ACBench-T2I points and 1.75 ACBench-Edit points, while 1024 × 1024 adds only 0.94 and 0.46 points at higher training cost.The study retains 512 × 512 as the default.Source: I Pure-Color Anchor Resolution · p. 22, Table 10 · p. 23

J Editing under Large Color Shifts

  • Paint-Anything is effective across editing color-shift buckets and strongest on the farthest quartile and near-complementary cases, while the base model degrades as required shifts grow.CompColor Distant under the hex-translated protocol shows a similar pattern for target-pair contrast.Source: J Editing under Large Color Shifts · p. 23, Table 11 · p. 23

K User Study

  • Across 1,200 color-preference judgments, Paint-Anything is preferred in 55%, tied in 32%, and loses in 13%; the non-tie win rate is 80.9%.Across 200 image-quality judgments, it wins 22%, ties 58%, and loses 20%; these descriptive results do not establish per-sample metric-human agreement or image-quality equivalence.Source: K User Study · p. 23

L Training and Evaluation Separation

  • ACBench-Edit sources come from an internal dataset separate from Paint-500K, and normalized benchmark prompt strings have no exact overlaps with training prompts.The benchmark prompts are independently constructed and use different formats from training captions.Source: L Training and Evaluation Separation · p. 23

M Additional Training-Data Examples

Related papers

Loading 2609.20816v2…