PaperlayerView exact revision on arXiv

Source-linked AI summary

GlyphPrinter: Region-Grouped Direct Preference Optimization for Glyph-Accurate Visual Text Rendering

Xincheng Shuai, Ziye Li, Henghui Ding, Dacheng Tao

arXiv:2603.15616v1cs.CV

TL;DR

Accurate glyph rendering remains difficult for complex and out-of-domain characters, and recognition-based rewards can miss localized glyph errors. GlyphPrinter addresses this with region-level preference data, R-GDPO, and Regional Reward Guidance, outperforming existing methods in glyph accuracy while balancing stylization and precision.

  • Problem

    Visual text rendering still has limited glyph accuracy for complex or out-of-domain characters, while recognition-based rewards can overlook fine-grained glyph errors.

  • Method

    GlyphPrinter uses the GlyphCorrector dataset, Region-Grouped DPO over inter- and intra-sample region preferences, and Regional Reward Guidance during inference.

  • Results

    GlyphPrinter outperforms existing text-rendering methods in glyph accuracy while maintaining a favorable balance between stylization and precision.

  • Takeaways & Limitations

    Region-level preference optimization better aligns rendered outputs with correct glyph regions while discouraging incorrect ones, with inference-time control over glyph accuracy.

Abstract

from arXiv · show

Generating accurate glyphs for visual text rendering is essential yet challenging. Existing methods typically enhance text rendering by training on a large amount of high-quality scene text images, but the limited coverage of glyph variations and excessive stylization often compromise glyph accuracy, especially for complex or out-of-domain characters. Some methods leverage reinforcement learning to alleviate this issue, yet their reward models usually depend on text recognition systems that are insensitive to fine-grained glyph errors, so images with incorrect glyphs may still receive high rewards. Inspired by Direct Preference Optimization (DPO), we propose GlyphPrinter, a preference-based text rendering method that eliminates reliance on explicit reward models. However, the standard DPO objective only models overall preference between two samples, which is insufficient for visual text rendering where glyph errors typically occur in localized regions. To address this issue, we construct the GlyphCorrector dataset with region-level glyph preference annotations and propose Region-Grouped DPO (R-GDPO), a region-based objective that optimizes inter- and intra-sample preferences over annotated regions, substantially enhancing glyph accuracy. Furthermore, we introduce Regional Reward Guidance, an inference strategy that samples from an optimal distribution with controllable glyph accuracy. Extensive experiments demonstrate that the proposed GlyphPrinter outperforms existing methods in glyph accuracy while maintaining a favorable balance between stylization and precision.

1. Introduction

Visual text rendering methods still struggle with glyph accuracy, particularly for complex or out-of-domain characters. GlyphPrinter addresses this gap by replacing recognition-based rewards with region-level preference optimization.

  • Complex Chinese characters and emojis remain challenging for advanced text-to-image text-rendering methods.
  • Existing methods use prompt-guided or glyph-image-guided conditioning, with some training on large-scale scene-text datasets to balance stylization and glyph accuracy.
  • Recognition models used as rewards can overlook incorrect glyphs, assigning inflated rewards to inferior rendered samples.
  • Image-level DPO can favor a sample with correct “123” over one with correct “45”, leaving localized glyph errors underlearned.For the condition “12345”, the former receives 60% overall accuracy versus 40% for the latter.
  • GlyphCorrector provides region-level annotations, while R-GDPO uses inter- and intra-sample preferences and RRG enables controllable glyph accuracy at inference.The dataset contains 7,117 images across 897 glyph conditions.

2. Related Works

Visual text rendering methods mainly rely on textual encoders or rendered glyph images to provide glyph conditions. These approaches require substantial text-image data and can generalize poorly to out-of-vocabulary characters, while recognition-based rewards may misjudge glyph errors.

  • Prompt-guided methods encode glyph information through character-level or multilingual text encoders.
  • Glyph-image-guided methods use rendered glyph images to improve generation of out-of-vocabulary characters.
  • Prompt-guided approaches require large amounts of text images to learn mappings from textual inputs to visual glyphs.
  • Existing text-rendering reinforcement-learning methods often use text-recognition rewards that are insensitive to glyph errors, inflating rewards for inferior images.

3. Preliminaries

The preliminaries describe classifier-free guidance, reinforcement-learning objectives, and DPO-based optimization for image generation. These formulations contrast external reward-guided sampling with preference-based learning and conditional score guidance.

  • Classifier-Free Guidance: Classifier-Free Guidance combines unconditional and conditional score functions, with guidance weight ω controlling alignment to the input condition.
  • Reinforcement Learning from Human Feedback: RL optimizes expected reward while regularizing the policy against a reference model with a KL-divergence term weighted by β.
  • Reinforcement Learning from Human Feedback: The RL objective evaluates generated image x_0 using a reward model under condition c, while p_ref and p_θ denote the reference and optimized models.
  • Direct Preference Optimization: DPO-based methods avoid expensive policy-gradient reinforcement learning and learn an implicit reward model through preference optimization.
  • Direct Preference Optimization: The flow-matching DPO loss compares winning and losing samples through model and reference velocity-prediction errors, with the partition function canceling for preference pairs.

4. Method: GlyphPrinter

GlyphPrinter fine-tunes a text-rendering model, then applies region-level preference optimization using GlyphCorrector and R-GDPO, followed by controllable Regional Reward Guidance at inference.

  • 4.1. Stage 1: Fine-Tuning on Text Images: GlyphPrinter fine-tunes a pretrained text-to-image model on multilingual synthetic and high-quality realistic text images to obtain a baseline.Text conditions are rendered into glyph images, encoded, and combined with prompt and noisy-image features.
  • 4.1. Stage 1: Fine-Tuning on Text Images: The attention mask restricts communication between each image text region and its corresponding glyph feature while retaining prompt-image and intra-modality attention.This design localizes glyph conditioning to the associated text block.
  • 4.2. Stage 2: Region-Level Preference Optimization: R-GDPO replaces image-level preference learning with region-level pairs, combining inter-sample comparisons across images and intra-sample comparisons within one image.The weighting parameter λinter controls the balance between the inter-sample and intra-sample objectives.
  • 4.2. Stage 2: Region-Level Preference Optimization: GlyphCorrector contains human annotations of correct and incorrect glyph regions from 7,117 candidate images generated for 879 prompt–glyph pairs.Annotators label localized errors such as extraneous or missing strokes after filtering candidates with low overall image aesthetics.
  • 4.3. Regional Reward Guidance: Regional Reward Guidance samples from an optimal distribution with an adjustable regularization weight that controls glyph accuracy during inference.The method uses the Stage 1 model as the reference model in the region-level optimization formulation.

5. Experiments

GlyphPrinter is evaluated across multilingual, complex, and out-of-domain text conditions using accuracy, aesthetic, alignment, and glyph-specific user-study measures. It outperforms existing methods in glyph accuracy while preserving a favorable balance between stylization and precision.

  • Evaluation Setup: The evaluation measures text accuracy, image aesthetic, text-image alignment, and glyph accuracy, with user study-based Glyph.Acc complementing recognition metrics.Two benchmarks assess performance across varied rendering scenarios.
  • Qualitative Results: GlyphPrinter generates more accurate complex Chinese glyphs, while competing methods produce extraneous or missing strokes.Qwen-Image and X-Omni fail on complex Chinese characters, whereas EasyText often adds or omits strokes.
  • Qualitative Results: GlyphPrinter preserves glyph fidelity under multilingual, emoji, stylized-font, and out-of-domain conditions.Prompt-guided methods may omit characters or fail on unseen fonts, while glyph-image-guided methods can lose structural details.
  • Quantitative Results: GlyphPrinter outperforms in text accuracy on both GlyphAcc-Multilingual and GlyphAcc-Complex benchmarks.The reported results also show more accurate glyphs than methods prone to extraneous or missing strokes.
  • Quantitative Results: GlyphPrinter achieves text-image alignment and image aesthetic results comparable to the same-base-model method, though some metrics are slightly below X-Omni and Qwen-Image.These comparisons reflect differences between the underlying fundamental models.
  • Ablation Studies: R-GDPO outperforms Stage 1, SFT, Mask-SFT, and single-component preference objectives in ablation studies.Region-level penalties and combined inter- and intra-sample preference usage improve results, while omitting these designs leaves glyph flaws.

6. Conclusion

GlyphPrinter combines region-level preference learning with controllable inference guidance to improve glyph accuracy. The method outperforms existing text-rendering approaches while balancing stylization and precision, but remains limited for very small characters and diverse aspect ratios.

  • Contributions: GlyphPrinter uses GlyphCorrector and Region-Grouped DPO to align outputs with correct glyph regions while discouraging incorrect ones.Regional Reward Guidance further enhances performance during inference with controllable glyph accuracy.
  • Conclusion: Extensive experiments show higher glyph accuracy than existing text-rendering methods while maintaining a favorable balance between stylization and precision.
  • Limitations: Very small characters remain less accurate because of VAE limitations, and fixed-resolution training limits diverse aspect-ratio outputs.The authors leave both issues for future work.

A.1. Construction of the Training Dataset

GlyphPrinter is trained with synthetic and realistic text-image data, then refined using GlyphCorrector region-level preferences. The dataset construction combines multilingual coverage, diverse styles, and iterative human correction.

  • Synthetic Dataset: The synthetic dataset overlays multilingual characters on 560K LAION background images.Languages include Latin, Chinese, Japanese, Korean, Arabic, and Thai.
  • Realistic Dataset: The realistic dataset contains 30K high-quality internet images annotated with OCR-based text recognition and VLM-generated captions.Placeholder tokens represent text from individual text blocks, and the collection mainly contains Chinese and English text.
  • GlyphCorrector: GlyphCorrector provides region-level preference pairs by comparing generated characters with synthetic renderings.Annotators perform cross-review and iterative correction until errors are resolved.
  • GlyphCorrector: GlyphCorrector groups multiple candidate images per glyph condition because a single preference pair may not cover all accurate regions of long or complex text.The motivation is illustrated with samples that correctly render different portions of the same condition.

B.2. Derivation of Eq. (8)

The derivation shows that the regional reward can be expressed through the log-ratio between the optimal and reference denoising policies, with value-function terms accounting for adjacent diffusion steps.

  • Objective reformulation: Eq. (8) is derived by reformulating the multi-step reward objective as a KL-regularized optimization over diffusion transitions.The objective combines stepwise regional rewards with divergence from the reference model.
  • Diffusion formulation: The diffusion process is represented using states, transitions, policies, actions, and a regional reward applied to masked model outputs.The reward is defined as R_m(M(x_{t-1}), c), where M selects elements within the given mask.
  • Optimal policy: The optimal denoising policy is proportional to the reference policy multiplied by the exponentiated optimal value advantage scaled by β.This yields the optimal distribution p*_θ for each denoising transition.
  • Reward recovery: The regional reward equals a β-scaled log ratio between optimal and reference transition probabilities plus a value-function difference.The regional partition function appears in the equivalent reward expression.
  • Derivation conclusion: The derivation concludes that the recovered reward expression has exactly the form of Eq. (8).An alternative derivation reaches the same equivalence after substituting the reward terms and applying Jensen’s inequality.

D. More Details of Benchmarks

The benchmarks evaluate multilingual and complex-glyph rendering using generated layouts and text conditions spanning multiple languages and image regions.

  • Benchmark construction: GlyphAcc-Multilingual contains 370 test cases spanning English, Chinese, Japanese, Korean, French, Vietnamese, and Thai.The benchmark targets multilingual text rendering.
  • Benchmark construction: GlyphAcc-Complex contains 97 test cases focused on complex Chinese characters.It complements the multilingual benchmark with a concentrated complex-glyph setting.
  • Layout generation: Each generated image is assigned one to four rectangular text bounding boxes with text content and an overall prompt containing rendering placeholders.The layout specification uses 1024×1024 images and ordered box coordinates.

E. More Details of Evaluation Metrics

Evaluation combines VLM-based text, aesthetic, and alignment metrics with a user study that directly assesses glyph accuracy.

  • Text accuracy: Text accuracy is measured using Normalized Edit Distance and sentence accuracy after VLM-based text recognition.The recognition prompt requests the generated text line by line.
  • Image-level evaluation: Image aesthetic and text-image alignment are scored by a VLM on a 1-to-100 scale with brief justifications.The evaluation covers visual quality and correspondence between the prompt and generated image.
  • User study: A user study with 20 volunteers evaluates aesthetic quality, text-image alignment, and glyph accuracy on a 1-to-10 scale.Participants compare rendered characters with the generated glyphs, and scores are averaged.

G. More Comparison Results

Across multilingual, complex, out-of-domain, and OneIG comparisons, GlyphPrinter is reported to improve glyph accuracy while balancing visual stylization and precision.

  • Multilingual and complex rendering: GlyphPrinter outperforms existing methods in glyph accuracy across multilingual and complex-glyph examples.Prompt-guided methods especially struggle with infrequent languages, while existing methods perform poorly on complex glyphs.
  • Multilingual and complex rendering: GlyphPrinter achieves a good balance between stylization and precision in the comparison cases.The comparison section presents this balance as a qualitative outcome.
  • Out-of-domain rendering: On out-of-domain inputs containing emojis and stylized-font characters, GlyphPrinter preserves fine-grained structural details of the condition.The comparison is reported in Fig. VI.
  • OneIG benchmark: GlyphPrinter achieves the best performance on both English and Chinese scenarios in the OneIG benchmark.The result is reported in Table I.
Loading 2603.15616v1…