PaperlayerView exact revision on arXiv

Source-linked AI summary

Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation

Hyojin Bahng, Seungjoo Yoo, Wonwoong Cho, David K. Park, Ziming Wu, Xiaojuan Ma, Jaegul Choo

arXiv:1804.04128v2cs.CV

TL;DR

Text-to-color methods have been limited in handling rich text and multimodal color meanings. Text2Colors generates multiple palettes from text and applies them to grayscale images through palette-guided colorization, with evaluations supporting multimodal palette generation and effective palette reflection. The approach is limited for unknown input tokens, which can produce washed-out grayish-brown palettes.

  • Problem

    Existing methods have limited support for rich text and typically produce only a single color or palette, despite color meanings being multimodal.

    Source: §1 · pp. 1–2

  • Method

    Text2Colors combines text-to-palette and palette-based colorization networks with the manually curated PAT dataset of 10,183 text-palette pairs.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    The CA-enabled model expresses multimodality, and qualitative evaluations show generated palettes are effectively reflected in colorized outputs.

    Source: §5.1 · p. 11, Part II: Colorization Comparisons · p. 13, §6 · p. 14

  • Takeaways & Limitations

    Text2Colors supports semantic palette generation from rich text and palette-guided grayscale colorization, including phrase-level inputs and unseen word combinations from the training vocabulary.

    Source: §1 · p. 2, §8.2 · pp. 19, 21

    The model struggles with unknown tokens not included in the training set, producing washed-out grayish-brown palettes.

    Source: §8.2 · p. 21

Abstract

from arXiv · show

This paper proposes a novel approach to generate multiple color palettes that reflect the semantics of input text and then colorize a given grayscale image according to the generated color palette. In contrast to existing approaches, our model can understand rich text, whether it is a single word, a phrase, or a sentence, and generate multiple possible palettes from it. For this task, we introduce our manually curated dataset called Palette-and-Text (PAT). Our proposed model called Text2Colors consists of two conditional generative adversarial networks: the text-to-palette generation networks and the palette-based colorization networks. The former captures the semantics of the text input and produce relevant color palettes. The latter colorizes a grayscale image using the generated color palette. Our evaluation results show that people preferred our generated palettes over ground truth palettes and that our model can effectively reflect the given palette when colorizing an image.

1 Introduction

Text2Colors addresses the limited expressiveness of prior text-to-color methods by generating multiple palettes from rich text and using them to colorize grayscale images. Its design combines multimodal palette generation, palette-guided colorization, and conditional generative modeling for semantic color expression.

1 Introduction

  • Unlike prior methods limited to single words or one recommended palette, Text2Colors supports rich text and multiple palette alternatives.This expands the stated range of creative expression through words.Source: §1 · p. 2
  • Text2Colors generates multiple plausible color palettes from rich single- or multi-word text and uses them to colorize grayscale images.The approach targets multimodal color perception and aims for outputs that reflect both palette colors and text semantics.Source: §1 · p. 2
  • The model introduces a deep architecture for natural-language palette generation, plausible palette-based colorization, and the manually curated PAT dataset.PAT contains 10,183 multi-word text and multi-color palette pairs.Source: §1 · p. 3

Color Semantics

  • Colors are linked to high-level semantic concepts, motivating text-based palette recommendation and automatic palette-guided colorization for artists, designers, and non-experts.Prior palette-query methods may map text to the colors of retrieved image content rather than to the colors implied by the text.Source: §1 · p. 1, Color Semantics · p. 3, Interactive Colorization · p. 4

Sequence-to-Sequence with Attention

  • Conditional generative adversarial networks provide the model’s conditioning framework, while attention helps relate parts of a text sequence to palette colors.The paper describes cGAN conditioning [24] and attention-based cross-modal relations in sequence-to-sequence models.Source: Conditional GANs · p. 4, Sequence-to-Sequence with Attention · p. 4

3 Palette-and-Text (PAT) Dataset

PAT is a manually curated dataset pairing rich text with five-color palettes, designed to support semantic and multimodal palette prediction. It expands beyond narrower color datasets through broad vocabulary and annotation-based refinement of noisy web data.

3 Palette-and-Text (PAT) Dataset

  • PAT contains 10,183 text and five-color palette pairs spanning direct color terms and concepts that evoke sets of colors.The dataset is intended to train semantically consistent palette prediction from textual inputs.Source: §3 · p. 4

Other Color Datasets

  • Compared with existing datasets, PAT covers 4,312 unique words and supports multi-word text mapped to multiple colors.The paper contrasts this scope with Munroe’s single-color corpus and Kobayashi’s 180-adjective, 1,170 three-color-palette dataset.Source: Other Color Datasets · pp. 4–5, §3 · p. 4

Data Collection

  • PAT was constructed by crawling 47,665 palette-text pairs from color-hex.com and removing non-alphanumerical and non-English words.The raw names included arbitrary phrases, random words, and typos that could hinder model performance.Source: Data Collection · p. 5
  • Four annotators retained pairs when at least three agreed that the text and palette were semantically matched, preserving some subjectivity while filtering noise.Spelling and punctuation errors were corrected after annotation.Source: Data Collection · p. 5
  • Text2Colors uses separate generators to infer a palette from text and colorize a grayscale image using palette information.During training, the colorization generator receives a palette extracted from the ground-truth image; at test time, the generated palette guides colorization.Source: Figure 4 · p. 6

4 Text2Colors: Text-Driven Colorization

Text2Colors maps rich text to diverse color palettes and uses those palettes to guide grayscale image colorization through two conditional GAN networks. Its architecture combines stochastic conditioning, attention, palette injection, and adversarial objectives.

4 Text2Colors: Text-Driven Colorization

  • Text2Colors uses conditional GANs for text-to-palette generation and palette-guided grayscale image colorization.TPN generates palettes from multi-word text, while PCN predicts colorizations from grayscale images and palettes.Source: §4 · p. 6

Objective Function

  • Huber losses encourage generated palettes and colorized images to remain close to ground truth, while adversarial training promotes plausible outputs.The TPN also uses KL divergence to smooth the conditioning manifold, and the PCN conditions its discriminator on both palette and image.Source: §4.1 · p. 7, Objective Function · p. 9, Networks Architecture · p. 9

4.1 Text-to-Palette Generation Networks (TPN)

  • Conditioning augmentation samples Gaussian latent variables from GRU-encoded text, enabling multiple plausible palettes for the same input.The sampled conditioning sequence is provided to the palette generator, while its mean conditions the discriminator.Source: §4.1 · p. 6, Networks Architecture · p. 8

Networks Architecture

  • The TPN generator uses a GRU decoder with attention to produce five Lab colors, with attention matching text conditions to successive palette colors.Each output color depends on the previous color, context vector, and previous hidden state; the five colors form one palette.Source: Networks Architecture · p. 8

4.2 Palette-based Colorization Networks (PCN)

  • The PCN receives a grayscale Lab lightness channel and a five-color Lab palette, outputting the image’s two chromatic channels.Palette features are spatially integrated into a U-Net-style colorization network to apply palette colors to suitable image regions.Source: Objective Function · p. 9, Networks Architecture · p. 9

4.3 Implementation Details

  • TPN is trained for 500 epochs on PAT, PCN for 100 epochs using palettes extracted from ground-truth images, and test-time colorization uses TPN-generated palettes.All networks use Adam with learning rate 0.0002; Fig. 6 illustrates multiple palettes from the same text and comparison with Heer and Stone [12].Source: §4.3 · p. 10, Figure 6 · p. 10

5 Experimental Results

The experiments evaluate palette generation and palette-based colorization across multimodality, semantic attention, and user preferences. Results indicate that the model generates diverse, text-sensitive palettes and that users prefer its palette and colorization outputs.

5.2 Analysis on Attention Outputs

  • Fig. 7 visualizes attention scores for two text inputs, showing which word each predicted color attends to.Green boxes encode attention scores, and dashed boxes identify the attended word.Source: Figure 7 · p. 11

5.1 Analysis on Multimodality and Diversity of Generated Palettes

  • The model's CA module enables multiple palettes for the same text, whereas the variant without CA and Heer and Stone's model produce zero multimodality.TPN palettes maintain matching colors, while higher diversity in Heer and Stone's model can include irrelevant colors.Source: §5.1 · p. 11

5.2 Analysis on Attention Outputs

  • Attention scores connect individual palette colors to particular words, with colors attending to “ghoul” becoming dark and gloomy and those attending to “fun” becoming bright.The attention mechanism helps reflect semantics from text inputs of varying lengths.Source: §5.2 · p. 11, Figure 7 · p. 11
  • Fig. 8 presents qualitative palette outputs showing subtle semantic-context differences, including for unseen text combinations.All combinations except the first column are unseen data.Source: Figure 8 · p. 12

5.3 User Study

  • The user study evaluates text–palette matching and palette-based colorization using 53 participants across two study parts.Generated palettes are compared with ground-truth palettes, while PCN outputs are compared with a state-of-the-art colorization baseline.Source: §5.3 · p. 12

Part I: Matching between Text and Generated Palettes

The first user-study part measures whether generated palettes match their text inputs better than ground-truth or prior-model palettes. TPN achieves a higher fooling rate than Heer and Stone's model.

Part I: Matching between Text and Generated Palettes

  • In a user study, participants completed 30 evaluations, each showing text input with generated and ground-truth palettes for one randomly selected item from 992 test data.Source: Part I: Matching between Text and… · p. 13
  • 56.2% FR for TPN exceeds Heer and Stone's model's 39.6%, indicating that generated palettes were often indistinguishable from or preferred over ground-truth palettes.The reported standard deviation for TPN is 12.7%.Source: Part I: Matching between Text and… · p. 13

Part II: Colorization Comparisons

The second user-study part compares PCN colorization with a modified state-of-the-art baseline using palette inputs. PCN scores higher across all surveyed questions and applies palettes using only five colors.

Part II: Colorization Comparisons

  • PCN colorizes images from five palette colors, whereas the modified baseline uses 313 bins of the ab gamut.The paper attributes this capability to conditional adversarial loss and feeding the palette into skip-connection layers.Source: Part II: Colorization Comparisons · p. 13
  • Fig. 10 compares colorization results with previous work [45] using a five-color palette beside the grayscale input.The passage states that PCN better applies the various colors included in the palette.Source: Figure 10 · p. 14
  • PCN scores higher than the baseline across all five palette-use questions, according to the statistics reported in Fig. 9.The survey uses a five-point Likert scale and evaluates how well the palette is used to colorize the grayscale image.Source: Part II: Colorization Comparisons · p. 14

6 Conclusions

The paper concludes that its generative model produces multiple palettes from rich text and uses them to colorize grayscale images. Evaluation supports plausible, multimodal palettes and effective reflection of palette colors in colorization, while future work targets broader tasks and dataset analysis.

6 Conclusions

  • The proposed model produces multiple palettes from rich text and colorizes grayscale images using those generated palettes.This conclusion summarizes the paper's central generation-and-colorization pipeline.Source: §6 · p. 14
  • Evaluation confirms plausible text-based palettes that incorporate color multimodality, while qualitative PCN results show that palette colors are reflected in colorization outputs.Future work includes broader color-recommendation tasks and detailed dataset analysis.Source: §6 · p. 14

7 User Study Samples

The user study evaluates Text2Colors in two parts: text-to-palette generation and palette-based colorization, with participant-facing data tuples shown in Fig. 11.

7 User Study Samples

  • The study evaluates both Text-to-Palette Generation Networks and Palette-based Colorization Networks using separate participant tasks.Fig. 11(a)–(b) shows how data tuples were presented in Parts I and II.Source: §7 · p. 18
  • Fig. 11 provides the user-study interface designs for the two evaluation parts.Source: Figure 11 · p. 18, §7 · p. 18

8 Text-to-Palette Generation Networks (TPN)

TPN generates palettes from text, with cGAN+Huber best matching global color distributions and producing diverse saturated colors. Additional results show semantic phrase sensitivity, success on sparse known vocabulary, and failures on unknown tokens.

8.1 Model Comparisons for Learning Global Color Distributions

  • 56.2% fooling rate and 0.2299 KL divergence make cGAN+Huber the strongest tested variant for matching ground-truth color distributions.Without Huber, fooling rate falls to 30.7%; Huber-only performs second best.Source: §8.1 · p. 18, Figure 12 · p. 19
  • cGAN+Huber preserves varied ground-truth colors and produces bright, highly saturated palettes instead of averaging them into desaturated outputs.Huber-only averaging causes slight desaturation, while the combined objective preserves color variation.Source: §8.1 · p. 19

8.2 Additional Results

  • For unseen phrases containing “love,” TPN produces darker palettes for inputs expressing negative attitudes toward love.The examples include “i thought i loved you” and “where did our love go.”Source: §8.2 · p. 19, Figure 14 · p. 20
  • When words appear in training, TPN can produce semantically related palettes even from sparse evidence, including “grapefruit,” seen only once.Generated palettes need not exactly match ground truth to remain perceptually reasonable.Source: §8.2 · pp. 19, 21, Figure 15 · p. 20
  • TPN fails on unknown tokens by repeatedly producing washed-out grayish-brown palettes, but handles new combinations of known words such as “bright life.”Unknown tokens are absent from training, whereas “bright life” combines separately observed words.Source: §8.2 · p. 21, Figure 16 · p. 21

9 Palette-Based Colorization Networks (PCN)

PCN colorizes grayscale images using palettes generated from text across natural-image and graphical-pattern datasets. The outputs reflect palette hints, can differ from natural colors, and transfer qualitatively to diverse shapes and textures.

9 Palette-Based Colorization Networks (PCN)

  • PCN colorizes grayscale inputs using text-generated palettes across CUB, ImageNet, and graphical-pattern datasets.The figures show grayscale inputs, text, generated palettes, colorized outputs, and original ground-truth images for comparison.Source: §9 · p. 21
  • Palette guidance can intentionally produce colors different from an image’s natural colors because the networks incorporate additional color hints.Source: §9 · p. 21

9.1 CUB-200-2011

9.2 ImageNet ILSVRC Object Detection

9.3 Graphical Pattern Images

  • On graphical patterns, PCN applies generated palettes to images with diverse shapes and textures despite training primarily on natural images.The qualitative results support transfer of the palette-based colorization model to another image domain.Source: §9.3 · p. 23, Figure 20 · p. 24, Figure 21 · p. 24

Related papers

Loading 1804.04128v2…