PaperlayerView exact revision on arXiv

Source-linked AI summary

TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Rendering

Hanshen Zhu, Yuliang Liu, Xuecheng Wu, An-Lan Wang, Hao Feng, Dingkang Yang, Chao Feng, Can Huang, Jingqun Tang, Xiang Bai

arXiv:2602.20903v3cs.CV

TL;DR

Visual Text Rendering remains difficult because OCR and MLLM evaluators largely miss fine-grained structural anomalies, making evaluation and RL rewards unreliable. TextPecker introduces a plug-and-play structure-aware RL strategy with composite rewards and character-level anomaly data. It consistently improves diverse generators, including Qwen-Image, where Chinese rendering gains 8.7% in semantic alignment and 4% in structural fidelity.

  • Problem

    Current OCR and MLLM evaluators largely fail to perceive fine-grained structural anomalies, limiting reliable VTR evaluation and reinforcement-learning optimization.

  • Method

    TextPecker combines a structure-aware composite reward with character-level annotated real artifacts and stroke-edited synthetic data for plug-and-play RL optimization.

  • Results

    8.7% semantic-alignment and 4% structural-fidelity gains are achieved for Chinese rendering on Qwen-Image, alongside consistent improvements across diverse generators.

  • Takeaways & Limitations

    TextPecker provides a structure-aware framework for improving structurally faithful visual text rendering across leading text-to-image generators.

  • Takeaways & Limitations

    The reward formulation assumes access to a robust assessor for fine-grained structural anomalies, while the required labeled data is scarce.

Abstract

from arXiv · show

Visual Text Rendering (VTR) remains a critical challenge in text-to-image generation, where even advanced models frequently produce text with structural anomalies such as distortion, blurriness, and misalignment. However, we find that leading MLLMs and specialist OCR models largely fail to perceive these structural anomalies, creating a critical bottleneck for both VTR evaluation and RL-based optimization. As a result, even state-of-the-art generators (e.g., Seedream4.0, Qwen-Image) still struggle to render structurally faithful text. To address this, we propose TextPecker, a plug-and-play structural anomaly perceptive RL strategy that mitigates noisy reward signals and works with any textto-image generator. To enable this capability, we construct a recognition dataset with character-level structural-anomaly annotations and develop a stroke-editing synthesis engine to expand structural-error coverage. Experiments show that TextPecker consistently improves diverse text-to-image models; even on the well-optimized Qwen-Image, it significantly yields average gains of 4% in structural fidelity and 8.7% in semantic alignment for Chinese text rendering, establishing a new state-of-the-art in high-fidelity VTR. Our work fills a gap in VTR optimization, providing a foundational step towards reliable and structural faithful visual text generation.

1. Introduction

Visual Text Rendering remains difficult because current OCR and MLLM evaluators overlook fine-grained structural defects, producing unreliable rewards for optimization. TextPecker addresses this bottleneck with structure-aware rewards and annotated data, improving diverse generators, including Qwen-Image.

  • VTR systems still produce distortion, blurriness, misalignment, and missing characters despite progress in text-to-image generation.
  • OCR models and MLLMs misinterpret flawed glyphs or ignore blurry regions, so they cannot reliably perceive fine-grained structural anomalies.Their semantic-recognition priorities can produce hallucinated corrections and dismiss low-confidence text.
  • Unreliable structural assessment yields inaccurate text scores and misleading rewards, hindering both VTR evaluation and reinforcement-learning optimization.
  • TextPecker replaces noisy OCR-based rewards with a plug-and-play composite reward jointly measuring semantic alignment and structural fidelity.Its structural term penalizes subtle glyph deformation and distortion without requiring generator architectural changes.
  • TextPecker’s hybrid dataset combines character-level annotations of authentic artifacts with stroke-edited synthetic data to expand structural-error coverage.
  • 8.7% semantic-alignment and 4% structural-fidelity gains are reported for Chinese rendering on Qwen-Image, establishing a new high-fidelity VTR state of the art.

2. Related Work

Prior VTR evaluation emphasizes textual accuracy and uses OCR or MLLMs, while structural-quality methods remain limited. TextPecker instead combines structure-aware recognition with reinforcement-learning rewards.

  • VTR assessment has predominantly focused on textual accuracy, often using standard OCR models or MLLMs as evaluators.
  • Existing accuracy-oriented methods address order mismatch and text-length imbalance through PNED and GNED normalization.
  • End-to-end MLLM evaluation can suffer from hallucination and inaccuracies, while prior structural assessors may be confined to single-character or document-analysis settings.
  • The framework samples candidate outputs, computes joint rewards, converts them into group-relative advantages, and optimizes the policy while enforcing KL proximity to a reference policy.
  • TextPecker moves beyond noisy OCR rewards by jointly quantifying semantic accuracy and structural fidelity with a perception-guided composite reward.

3. Methodology

TextPecker combines structure-aware reward modeling with RL and a structurally annotated, synthetically augmented dataset to improve visual text rendering. Its composite reward jointly measures semantic alignment and structural fidelity while the data pipeline expands coverage of character-level anomalies.

  • RL Preliminaries: TextPecker focuses GRPO-based optimization for text-to-image rendering, whose deterministic flow-matching models require injected stochasticity through Flow-GRPO.Flow-GRPO converts deterministic integration dynamics into a stochastic differential equation, while GRPO uses group-wise relative advantages and intra-group reward normalization.
  • Reward Modeling: Prior string-level rewards can overestimate quality because OCR or MLLM recognition lowers edit distance by hallucinating corrections or omitting distorted regions.The resulting biased rewards hinder effective optimization of structurally faithful text.
  • Reward Modeling: TextPecker replaces noisy OCR signals with a composite reward that jointly optimizes semantic alignment and structural fidelity across diverse generators.The structure-aware module identifies fine-grained defects such as missing or spurious strokes and supplies penalties for subtle glyph distortions.
  • Structure-aware Reward Functions: Structural Quality Score penalizes the proportion of anomalous characters, amplifying rare but severe errors with a scaling factor ω > 1.The score counts anomalous characters among generated characters and clips the result to [0, 1].
  • Structure-aware Reward Functions: Semantic Alignment Score uses Hungarian word matching with Normalized Edit Distance and penalties for unmatched generated or target words.The score addresses order mismatch and treats extraneous, repeated, or missing content as errors before clipping the result to [0, 1].
  • Structural Perceptive Data Construction: The dataset pipeline generates text-rich images, annotates character-level anomalies, and augments coverage using stroke deletion, swapping, and insertion.Chinese characters are represented as ordered stroke sequences, edited with sampled stroke points, and rendered alongside canonical text across backgrounds and layouts.

4. Experiments

Experiments evaluate TextPecker as a structural-anomaly-aware evaluator and reward across recognition, benchmark, qualitative, and ablation settings. Results show weak anomaly perception in existing models and consistent gains from TextPecker across generators and languages.

  • Recognition Evaluation: Existing MLLMs and specialist OCR models show near-total failure on Text Structural Anomaly Perception (TSAP).Their semantic-recognition training and limited exposure to generated structural anomalies hinder structural fidelity assessment.
  • Recognition Evaluation: Box-level recognition recall is substantially lower than image-level recall for existing models, limiting fine-grained local assessment.The limitation affects demanding tasks such as controllable local text editing and translation.
  • Recognition Evaluation: TextPecker achieves high TSAP F1 and recall while improving Canonical Text Recognition (CTR), with Qwen3-VL strongest for Chinese and InternVL3 strongest overall at box level.These results support the value of the box-level structure-aware dataset.
  • RL for VTR: +38.3% Sem. and +31.6% Qua. are achieved by Flux.1[dev] over its base model, while TextPecker improves results across four benchmarks, three base models, and English and Chinese tasks.On GenTextEval, the gain over the OCR-reward baseline is +11.7% Sem.
  • RL for VTR: +14.3% on OneIG, +7.4% on LongText, and +8.7% on GenTextEval are reported for Qwen-Image on Chinese rendering over prior SOTA.Compared with the OCR-reward baseline, Qua. increases by +2.0% and Sem. by +2.3%.
  • Qualitative Comparisons: TextPecker-based RL produces cleaner, more aligned text than vanilla Qwen-Image and OCR-reward optimization in challenging paper and menu examples.The qualitative comparisons highlight reduced distortion and improved legibility.
  • Data Composition: Training on annotated data substantially improves TSAP and English recognition, but synthesized augmentation is needed to resolve degraded Chinese recognition.The data-composition study attributes the Chinese difficulty to the complexity of structural anomalies in Chinese characters.
  • Ablation Studies: Combining TextPecker recognition with structural quality reward achieves the best overall performance in reward-component analysis.A structure-unaware OCR model with Pairwise Matching improves Sem. by 4.2% but leaves structural quality stagnant; replacing it with TextPecker gives Sem. +5.8% and Qua. +0.8%.

5. Conclusion

The conclusion identifies fine-grained structural-anomaly perception as the bottleneck in VTR evaluation and RL optimization and presents TextPecker as a unified remedy. Across leading generators, the framework improves semantic alignment and structural quality, including on Qwen-Image.

  • 5. Conclusion: TextPecker couples a standardized structural-anomaly-aware evaluator with RL rewards for semantic alignment and structural quality.The framework is plug-and-play and designed for text-to-image generators.
  • 5. Conclusion: Structure-aware rewards shift generation toward fewer off-target strings, reduced blur and distortion, and improved alignment.The conclusion presents these changes as the observed generation behavior under TextPecker rewards.
  • 5. Conclusion: TextPecker provides tools for rigorous VTR evaluation and post-training enhancement toward structurally faithful visual text rendering.The paper characterizes this as a foundational step within the supported VTR scope.

(Supplementary Materials)

Supplementary materials add visualizations of manually annotated and synthesized anomalies, TextPecker evaluations, and qualitative comparisons for Flux.1[dev].

  • Supplementary Materials: Additional figures visualize annotated structural anomalies, synthetic anomalies, TextPecker evaluation samples, and Flux.1[dev] RL variants.The materials cover diverse generative models and qualitative RL comparisons.

B. Additional Ablation Studies

Additional ablations isolate the contributions of Pairwise Matching, TextPecker recognition, and structural quality reward. The full reward design achieves the strongest combined semantic and structural performance.

  • B. Additional Ablation Studies: Pairwise Matching with structure-unaware OCR improves semantic alignment by 1.2% but structural quality by only 0.3%.This indicates limited structural benefit without structural perception.
  • B. Additional Ablation Studies: Replacing the OCR model with TextPecker yields Sem. +1.6% and Qua. +4.6%.The result demonstrates gains across both evaluation dimensions from the structure-aware assessor.
  • B. Additional Ablation Studies: Adding structural quality as an auxiliary reward gives further Sem. +0.8% and Qua. +0.3%, producing the best overall performance.The ablation supports synergy among the full TextPecker reward components.

C. Additional Generalization Results

TextPecker is evaluated across Gemini-2.5-flashimage renderings and multiple Chinese benchmarks, showing consistent robustness while exposing failures on extreme stylization.

  • Robustness evaluation: TextPecker remains consistent across normal, extreme-stylization, and low-contrast Gemini-2.5-flashimage renderings.The evaluation uses TSAP-F1 and CTR-R across the three conditions.
  • Robustness evaluation: Its main failures occur with extremely stylized fonts, where artistic deformations distort canonical glyph structure and blur style–error boundaries.
  • Benchmark evaluation: The additional Chinese evaluation compares OneIG, LongText, and GenTextEval using multi-reward measurements.The reward setting includes OCR, semantic, structural-quality, PickScore, and aesthetic rewards.

D. Additional Results on RL for VTR

Additional RL experiments strengthen the optimization baseline with improved training stabilization and multi-reward regularization, while qualitative comparisons examine the resulting English and Chinese renderings.

  • Enhanced RL baseline: Flow-GRPO-Fast and related GRPO techniques are added to improve the efficiency and stability of VTR optimization.
  • Enhanced RL baseline: Velocity-based KL regularization replaces state-based regularization to stabilize training dynamics and mitigate over-optimization.
  • Qualitative comparisons: Qualitative comparisons cover Flux.1[dev] and RL baseline settings for both English and Chinese text rendering.
  • Multi-reward regularization: A single text-rendering reward can degrade aesthetic and image-quality performance, motivating multi-reward regularization.
  • Quantitative comparisons: The supplementary experiments include Chinese and English benchmark comparisons under the multi-reward setting.

E.2. Details on Synthetic Data Augmentation

The synthetic-data pipeline expands structural-anomaly coverage for Chinese text and supports diverse evaluation through varied fonts, prompts, generated images, and real-only test splits.

  • Motivation: Manual annotations generalize poorly to unseen structural anomalies, especially for Chinese characters with 2D structure and roughly 8,000 common characters.
  • Synthetic augmentation: The enhanced SynthTIGER renderer adds complex image-level layouts and a Structural Anomaly Construction engine for diverse Chinese errors.
  • Rendering design: The rendering engine uses a 976-type font pool while limiting postprocessing and geometric transformations to preserve structural clarity.
  • Evaluation data: The structural-perception test set includes box- and image-level anomaly labels, with additional evaluation on a real-only split.
  • RL prompt construction: RL prompts cover diverse text lengths and content, drawing English data from TextAtlas5M and Chinese data from WanJuan1.0 with generated font-style descriptions.
  • Benchmark construction: GenTextEval combines English and Chinese prompts from multiple sources and adds Chinese prompts because Chinese benchmark availability is limited.
  • Evaluation protocol: Identical prompting templates are used for TextPecker and other MLLM baselines to support consistent comparisons.

G. Additional Implementation Details

The implementation uses Flow-GRPO for RL-based VTR optimization, model-specific sampling configurations, and an asynchronous evaluator with negligible inference overhead.

  • RL framework: Flow-GRPO is used for all RL-based VTR optimization experiments, following the stable repository version available during the experiments.
  • Model configurations: SD3.5-M, Flux.1[dev], and Qwen-Image use different sampling steps, noise levels, guidance scales, KL ratios, and LoRA configurations.
  • Benchmark statistics: Rendered-text statistics compare text lengths in words for English and characters for Chinese across several benchmarks.
  • Evaluation runtime: The evaluator runs asynchronously during RL training, adding negligible overhead without affecting inference latency.
  • Evaluation runtime: On SD3.5-M, 100 RL steps take 5.52 h with TextPecker versus 5.40 h with PPOCRv5.
  • Enhanced baseline: The enhanced RL baseline combines Flow-GRPO-Fast, GRPO-Guard, Velocity KL loss, and multi-reward regularization.
  • Enhanced-baseline configurations: Enhanced-baseline configurations use model-specific sampling schedules, SDE windows, guidance scales, learning rates, KL ratios, and clipping ranges.

I. Limitations

The paper identifies several scope and applicability limitations for TextPecker, including dependencies on font data, language coverage, downstream-task validation, and artistic text generation.

  • Stroke-level font data is required for anomaly synthesis, restricting coverage to standard fonts rather than artistic or proprietary typefaces.
  • The current system supports only Chinese and English text rendering, leaving efficient multilingual extension for future work.
  • TextPecker’s downstream effectiveness for text translation and local text editing remains unvalidated.The evaluator theoretically supports these tasks through box-level perception, but evaluation and RL optimization are left for future work.
  • Artistic text generation remains challenging because deliberate structural modifications make a single standard or ground truth difficult to define.Continuously evolving artistic designs also conflict with the framework’s structural-consistency objectives.
Loading 2602.20903v3…