PaperlayerView exact revision on arXiv

Source-linked AI summary

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

Deyuan Liu, Yihao Hu, Jingxuan Zhang, Xingying Li, Jun Xie, Jiacheng Liu, Jungang Li, Yu Huang, Xuanyi Liu, Yue Ding, Zecheng Wang, Lei Zhao, Mingda Wang, Zhenglin Cheng, Peng Sun, Tao Lin

arXiv:2610.09823v1cs.CVcs.AI

TL;DR

Dense visual text requires generators to reproduce exact strings across multiple regions, and existing evaluation must test sustained performance beyond short strings. UltraText Bench evaluates prompt-only generation with bilingual prompts and structured references; results show that clarity, fidelity, and performance across workloads can diverge.

  • Problem

    Dense visual text requires exact, legible strings across multiple regions, so evaluation must test sustained performance beyond short-string rendering.

    Source: §1 · p. 1

  • Method

    UltraText Bench evaluates prompt-only generation using bilingual prompts with exact target strings and structured per-region references, scored by Q-Judger across four dimensions.

    Source: §1 · p. 3, §3.1 · p. 7

  • Results

    Across model configurations, quality dimensions and workload results diverge; Z-Image-Turbo has 14.76 points lower Fidelity and 3.81 points higher Clarity than Z-Image-Base under reported settings.

    Source: §5.2 · p. 13

  • Takeaways & Limitations

    Dense-text assessment benefits from separately tracking fidelity, clarity, spatial quality, and scene quality across workload levels and languages.

    Source: §1 · p. 3

    Results use model-specific standard settings that differ in resolution, aspect ratio, and API behavior; controlled comparison of these factors remains future work.

    Source: §6 · p. 14

Abstract

from arXiv · show

Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal different strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512's English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores. Repository: https://github.com/LINs-lab/UltraText_Bench.

1 INTRODUCTION

UltraText Bench targets dense, multi-region visual text, where a convincing image can still omit or misrender requested content. It pairs bilingual prompts with structured references and evaluates whole scenes across four dimensions; results across 24 configurations vary by dimension and workload.

1 INTRODUCTION

  • 432 prompts cover 24 real-world categories, three difficulty levels, and English and Chinese, with four to twelve text regions per prompt.Each record pairs exact target strings with structured per-region references for content, placement, and visual attributes.Source: §1 · pp. 2–3
  • Q-Judger assesses each generated image against the complete reference, reporting text fidelity, clarity, spatial quality, and scene quality.Accuracy and completeness form text fidelity, while position and layout form spatial quality.Source: §1 · p. 3
  • Across 24 model configurations, clarity and fidelity diverge between Base and Turbo variants, and performance varies across workload levels and languages.The reported Z-Image-Turbo comparison shows higher clarity but lower fidelity than Z-Image-Base.Source: §1 · p. 3
  • Ten participants took part in human evaluation of the automatic scores.Source: §1 · p. 3

2 RELATED WORK

Related work spans methods for binding text to image regions, benchmarks with varied text loads and evaluation protocols, and rubric-based automatic judges. UltraText Bench situates itself in prompt-only dense-text evaluation with uniform per-region references and whole-image scoring.

2.1 VISUAL TEXT GENERATION METHODS

  • Visual-text methods use glyph or layout conditioning, character-aware and multilingual modeling, and approaches for dense multi-region generation and few-step synthesis.Inputs range from prompts alone to explicit glyph and layout controls; layout prediction errors can propagate to generation.Source: §2.1 · pp. 3–4

2.2 TEXT RENDERING BENCHMARKS AND DATASETS

  • Benchmarks cover short-string fidelity, dense and long text, video, and broader generation or editing tasks, but their inputs and evaluation protocols differ.Editing benchmarks are not directly comparable to prompt-only generation because they provide a source image.Source: §2.2 · pp. 4–6
  • UltraText Bench combines prompt-only generation, per-region references, and 24 text-bearing scene categories, with image-level scores and region-level attribution left for future work.Its six-field annotation is reserved for evaluation and stays consistent across scenes, languages, and levels.Source: §2.2 · p. 6

2.3 AUTOMATIC EVALUATION AND JUDGE RELIABILITY

  • Automatic evaluation includes global preference metrics, OCR-based fidelity measures, text-quality predictors, and rubric-based VLM judges, each with distinct limitations.OCR may introduce recognition errors; VLM scores can be affected by evaluator version, prompting, text-reading ability, and visual-quality bias.Source: §2.3 · pp. 6–7

3 ULTRATEXT BENCH: BENCHMARK DESIGN

UltraText Bench evaluates text fidelity, clarity, spatial quality, and scene quality across dense text-bearing scenes, workloads, and languages. Its prompt-only task pairs target strings with per-region references, while its scene taxonomy and sampling design span varied rendering challenges.

3 ULTRATEXT BENCH: BENCHMARK DESIGN

  • UltraText Bench examines how text fidelity, clarity, spatial quality, and scene quality vary across dense text-bearing scenes, workloads, and languages.Source: §3 · p. 7

3.1 TASK FORMULATION

  • The prompt-only task supplies all target strings and evaluates generated images against structured per-region references, with whole-image scores rather than region-level scores or transcriptions.Each region records its target string, grid position, relative size, type, carrier, and importance; an example signage record contains 423 GT characters across four regions.Source: §3.1 · p. 7

3.2 CONSTRUCTION RATIONALE

  • 432 prompts cover 24 categories, three difficulty levels, and equal English and Chinese splits, with three prompts in each category-by-level-by-language cell.Text load and region count increase together across levels, so the levels do not isolate their individual effects.Source: §3.2 · p. 7
  • The taxonomy groups 24 categories into six domains by rendering challenge and typical layout; density is qualitative, and layouts are examples rather than required templates.Source: Table 2 · p. 8
  • Signage categories cover challenges including perspective and weathered surfaces, curved labels, font hierarchy, and billboard scale and viewing angles.Source: §3.2 · p. 8

B. Documents & Print

The document-and-print categories represent varied document structures and text layouts.

B. Documents & Print

  • Articles are characterized by paragraph flow and alignment, while newspapers add columns and headline hierarchy.Source: B. Documents & Print · p. 8
  • Letters organize a salutation, body, and closing; resumes emphasize sections and bullet alignment.Source: B. Documents & Print · p. 8
  • The listed document layouts range from single-column articles and letters to multi-column newspapers and two-column resumes.Source: B. Documents & Print · p. 8

C. Commercial

Commercial scenes include menus, receipts, invoices, and product packaging, each associated with distinct text and layout challenges.

C. Commercial

  • Menus and invoices emphasize tabular layouts, including menu item-price alignment and invoice numeric fields and table alignment.Source: C. Commercial · p. 8
  • Receipts present narrow lines and exact totals in a single-column layout.Source: C. Commercial · p. 8
  • Product packaging combines perspective with ingredient lists in a multi-panel layout.Source: C. Commercial · p. 8

D. Digital Interfaces

Digital-interface categories cover webpages, slides, social-media interfaces, and dashboards, with layouts ranging from stacked blocks to grids.

D. Digital Interfaces

  • Webpages combine navigation, content, and a sidebar in a multi-panel layout; dashboards arrange KPI cards and chart labels in a grid.Source: D. Digital Interfaces · p. 8
  • Slides organize titles and bullets in stacked blocks, while social-media scenes combine interface text and hashtags in a card or feed.Source: D. Digital Interfaces · p. 8

E. Structured Data

The structured-data examples show how scene types are paired with characteristic text-layout requirements.

E. Structured Data

  • Schedules are characterized by time-column alignment, while forms require label-field alignment.Both are classified as tabular scenes.Source: E. Structured Data · p. 8
  • Certificates emphasize formal typography and alignment, with centered blocks as their layout.Their typography difficulty is rated Low–Med.Source: E. Structured Data · p. 8
  • Code scenes call for monospace text and indentation, with indented lines and High difficulty.Source: E. Structured Data · p. 8

F. Creative & Special

This section describes a benchmark built from varied text-bearing scenes, language-specific workloads, and per-region ground truth. Its reviewed prompts span increasing text and region loads, with distributions that differ across English and Chinese.

F. Creative & Special

  • The creative and special categories include captions, dialogue, comics, and infographics with distinct text-placement demands.Examples include overlay legibility, alternating speech blocks, scattered bubbles, and freeform label placement.Source: F. Creative & Special · p. 8

3.3 SCENE TAXONOMY: 24 CATEGORIES IN 6 DOMAINS

  • The benchmark organizes 24 categories across six domains, including photographs, documents, and digital interfaces.Table 2 groups categories by carrier, density, and layout.Source: §3.3 · p. 8

3.4 DIFFICULTY STRATIFICATION

  • Difficulty levels L1, L2, and L3 assign increasing text and region workloads, rather than disjoint character-count bins.English design bands are 250–600, 600–1200, and 1200+ characters; Chinese bands are 350–600, 550–900, and 600+ characters.Source: §3.4 · pp. 8–9

3.5 BILINGUAL STRATEGY

  • English and Chinese prompts are independently written and culturally adapted, so their score comparison does not isolate language alone.The splits balance category and level counts but use different text-load bands.Source: §3.5 · p. 9

3.6 PROMPT CONSTRUCTION AND GROUND TRUTH

  • Each prompt includes exact target strings and structured region references for position, relative size, text type, carrier, and importance.The 3 × 3 grid encodes position; size is qualitative, and importance does not enter the scoring formula.Source: §3.6 · p. 9

3.7 QUALITY ASSURANCE

  • All 432 records undergo manual review and automated checks for target inclusion, annotations, and assigned workload bands.Each region is checked for an identifier and all six annotated fields.Source: §3.7 · pp. 9–10

3.8 DATASET STATISTICS

  • English mean character load rises from 489.90 at L1 to 2,386.68 at L3, while Chinese rises from 394.62 to 743.62.Mean region counts also rise, from 4.67 to 9.38 in English and 4.93 to 8.40 in Chinese; Table 3 reports per-prompt means and ranges.Source: §3.8 · p. 10, Table 3 · p. 10

4 EVALUATION FRAMEWORK

The framework compares generated images with complete structured references using a shared Q-Judger rubric. It reports four dimensions and includes human evaluation, while selected examples illustrate disagreements between the judge and visible image details.

4 EVALUATION FRAMEWORK

  • Q-Judger scores each generated image against its complete structured reference, with failed evaluations left unscored and reducing coverage.All target strings reach the judge without truncation.Source: §4 · p. 10, §4.1 · p. 10

4.1 FOUR-DIMENSION SCORING

  • Six raw ratings are grouped into fidelity, clarity, spatial quality, and scene quality, with 60% and 30% weights assigned to fidelity and clarity.Spatial and scene quality each contribute 5%; the raw scores can distinguish correctness from completeness.Source: §4.1 · pp. 10–11

4.2 HUMAN EVALUATION

  • Ten participants took part in human evaluation, while selected examples show local judge disagreements without estimating their frequency.Quantitative inter-rater and human–judge agreement statistics are not reported.Source: §4.2 · p. 11

5 EXPERIMENTS

Across the evaluated configurations, GPT Image 2 [Low] leads the reported Composite, while results vary across dimensions, difficulty levels, and language-specific prompt sets. Comparisons are descriptive and depend on the reported model configurations and aggregation.

5.1 EXPERIMENTAL SETUP

  • The study compares 24 model configurations using standard sampling settings and descriptive prompt-macro means, without confidence intervals.Images from a shared prompt are related observations, so close comparisons require uncertainty estimates across prompts.Source: §5 · p. 11, §5.1 · pp. 11–12, §5.2 · p. 12

5.3 STRONG MODELS AND EVALUATION COVERAGE

  • 99.35 Composite: GPT Image 2 [Low] leads, with 1,440 of 1,723 valid images receiving 100 on all six raw dimensions.Its EN and ZH scores remain high across all three levels.Source: Table 5 · p. 12, §5.3 · p. 13

5.2 BENCHMARK RESULTS

  • Z-Image-Turbo scores 74.41 Clarity and 40.75 Fidelity, while Boogu-Image-0.1-Base scores 75.77 Fidelity and 78.31 Clarity.Qwen-Image-2512 scores 79.89 Clarity and 59.30 Fidelity, illustrating that clearer rated text need not preserve requested content best.Source: Table 5 · p. 12, §5.2 · p. 12
  • Turbo Fidelity is lower than Base by 14.76 points for Z-Image, 7.62 for LLaDA-Image, and 9.50 for Boogu-Image-0.1.Clarity rises by 3.81 and 7.37 points for Z-Image and LLaDA-Image, but falls by 13.45 for Boogu-Image-0.1; standard settings prevent attributing changes to acceleration alone.Source: Table 5 · p. 12, §5.2 · p. 13
  • Qwen-Image-2512’s EN Composite falls from 86.50 at L1 to 42.86 at L3, though difficulty-level trends are not universal.Levels group different prompts with changing text loads and region counts, so they do not isolate length effects.Source: Table 5 · p. 12, §5.2 · p. 13
  • At L3, Boogu-Image-0.1-Base scores 57.67 in EN and 83.23 in ZH, while GPT Image 1.5 [High] scores 77.21 and 18.22.These independently written prompt sets also differ in average text load: 2,386.68 GT characters in EN and 743.62 in ZH.Source: Table 5 · p. 12, §5.2 · p. 13

5.3 STRONG MODELS AND EVALUATION COVERAGE

  • GPT Image 2 [Low] leads all four reported means; Boogu-Image-0.1-Base remains the open-weight leader under the reported alternative weightings.Close rankings depend on tradeoffs: Qwen-Image-2512’s default Composite is 67.96 versus 67.79 for Boogu-Image-0.1-Turbo, which has higher Fidelity.Source: §5.3 · p. 13

6 DISCUSSION

Dense text poses character-level, multi-region demands distinct from ordinary image quality, while the discussion positions UltraText Bench alongside related benchmarks and identifies interpretation limits.

6 DISCUSSION

  • Text is discrete and compositional: a single wrong character can invalidate a word or code, while dense prompts spread model capacity across many local text regions.The proposed mechanisms may contribute to differences among text fidelity, layout, and scene quality, but their causal effects remain untested here.Source: §6 · p. 13
  • Structured annotations could support region-level diagnosis and feedback on character errors, omissions, placement, and scene integration for reward-based or preference-based training.Importance could also weight errors in a shop name more heavily than those in a footnote, though it does not enter the current scoring formulas.Source: §6 · pp. 13–14
  • UltraText Bench complements STRICT’s single-sequence plain-page setting with 4–12 regions across 24 categories, while focusing on prompt-only generation unlike OCRGenBench.InfoTextBench is the closest dense bilingual comparison, but its target-text items differ from UltraText Bench’s regions, preventing direct count comparisons.Source: §6 · p. 14
  • Comparisons warrant caution because model settings differ, means lack confidence intervals, and language and difficulty contrasts do not isolate language or text-load effects.The judge returns image-level scores, does not provide region-level verification, and selected cases show visible errors despite maximum ratings.Source: §6 · p. 14
  • Future work will explore region-level evaluation and structured annotations as training signals, and extend evaluation to video text persistence and temporal consistency.The proposed video extension includes generation and editing systems.Source: §6 · p. 14

7 CONCLUSION

This section summarizes UltraText Bench’s evaluation of dense bilingual text using complete region references to assess fidelity, clarity, placement, and scene integration. It also explains how image scores are validated and aggregated, including the distinction between strict-success quality means and planned-row coverage, and the limits of what the references and aggregates establish.

7 CONCLUSION

  • The benchmark is designed to assess sustained text reproduction, placement, and integration across workloads and languages, rather than fidelity alone.Its comparisons cover 24 model configurations.Source: §7 · p. 14

A DATASET CONSTRUCTION AND ANNOTATION

  • The 432 prompts were human-reviewed and independently authored in English and Chinese across 24 categories, three difficulty levels, and two languages.Automated checks verified character-load bands, region fields, and verbatim inclusion of each target string; OCR was not a release gate or ranking signal.Source: A DATASET CONSTRUCTION AND ANNOTATION · p. 20

A.1 ANNOTATION SCHEMA

  • Each region reference records exact text and six attributes, while the generator receives only the natural-language prompt.Position uses a coarse nine-cell grid without bounding boxes or within-cell ordering; importance does not affect scoring, and lower-importance regions cannot be ignored.Source: §A.1 · p. 21

A.2 CATEGORY DEFINITIONS

  • The 24 categories span scene families in which text is supplied for reproduction, so prices and other figures need not be inferred or calculated.The categories include signage, print, digital interfaces, structured data, and creative scenes.Source: §A.2 · p. 21

B INPUT EXAMPLES AND DIFFICULTY

  • The English L1 sign example has four regions and 423 GT characters, while its independently authored Chinese counterpart has five regions and 362 characters.The Chinese scene adds a window sticker and adapts details such as prices to Chinese usage; its strings are targets for character-for-character reproduction.Source: §B.1 · p. 23, §B.3 · p. 24

B.4 THREE LEVELS WITHIN ONE SCENE FAMILY

  • The newspaper examples illustrate progression across difficulty levels, but their differing content and layouts mean they do not isolate text length as the sole change.A grid cell can contain multiple regions without defining pixel boundaries between them.Source: §B.4 · p. 24

C.1 EVALUATION PROMPT TEMPLATE

  • The Q-Judger rubric compares every referenced region with the generated image and scores text correctness, completeness, readability, position, layout, and scene integration.The reference is supplied as evaluation metadata alongside the image, not as an extra layout input for the generator.Source: §C.1 · p. 26

C.2 REFERENCE PAYLOAD AND RESPONSE VALIDATION

  • Only strict successful judge responses enter quality means; missing, invalid, or failed rows remain in coverage denominators and receive no imputed score.A valid response must be one JSON object with exactly six integer scores from 0 to 100; failures carry null scores and a typed status.Source: §C.2 · p. 27, D SCORING AND AGGREGATION · p. 29

C.3 EXECUTION WORKFLOW

  • The bilingual execution plan keeps all 1,728 planned rows per model configuration in coverage accounting, while failed rows receive no quality score.Rows become successful only with a valid artifact and strict six-key response; a full language split has 864 images.Source: §C.3 · p. 28

D SCORING AND AGGREGATION

  • F = (TA + TC)/2, C = TR, S = (PC + LQ)/2, and Q = SI; the composite has no pass threshold, so raw dimensions remain important.A high TC score alone does not establish correct reproduction, and the composite can remain 70 even with TA=0 if the other five scores are 100.Source: D SCORING AND AGGREGATION · p. 29

D.1 WORKED EXAMPLE FROM A RECORDED RESPONSE

  • 92.0 is the stored composite for the worked example, after its image-level composite is rounded to one decimal before aggregation.The recorded raw scores yield F=90, C=95, S=92.5, and Q=98; the computed composite is 92.025.Source: §D.1 · pp. 29–30

D.2 QUALITY MEANS AND COVERAGE: A NUMERICAL EXAMPLE

  • 37.50% image coverage and 100.00% prompt coverage in the example show that coverage and quality answer different questions.Prompt-macro aggregation weights each represented prompt equally, while the image mean weights prompts according to their successful image counts; bilingual leaderboard means average EN and ZH prompt-macro means equally.Source: §D.2 · p. 30

D.3 SCOPE OF THE REPORTED AGGREGATES

  • Table 5’s first five score columns are equal-language means of EN and ZH prompt-macro aggregates, while its six level columns retain language-specific composites.Unavailable cells are not zero scores, and complete-reference input plus a valid response does not verify that every region was assessed correctly.Source: §D.3 · p. 30

E.1 WHAT THE REGION REFERENCE CONTRIBUTES

  • Structured references link each requested string to its intended role, location, and visual carrier, but the protocol reports no separate region-level metric.PC captures coarse position, while the grid does not encode within-cell ordering or all spatial relations; the six reference attributes guide image-level judgments but are not score weights.Source: §D.3 · p. 30, §E.1 · p. 31

E.2 BENCHMARK VALUE AS MODEL CAPABILITIES IMPROVE

Model quality varies across workloads, and different dimensions distinguish configurations with similar composite scores. Qualitative examples, category galleries, released records, and human-evaluation discussion show how results can be inspected and where the evidence remains bounded.

E.2 BENCHMARK VALUE AS MODEL CAPABILITIES IMPROVE

  • 86.50 to 42.86: Qwen-Image-2512’s EN Composite falls from L1 to L3, showing substantial workload variation across configurations.Qwen-Image-2512 and Boogu-Image-0.1-Turbo have Composites of 67.96 and 67.79, but Qwen has higher Clarity (79.89 versus 64.86) and Boogu higher Fidelity (66.27 versus 59.30).Source: §E.2 · p. 31

F QUALITATIVE EXAMPLES AND DIAGNOSTIC READING

F.1 CATEGORY GALLERIES

  • Category galleries show one selected output per taxonomy category at L1 and L2, with the L3 gallery in the main text, illustrating scene diversity.The galleries are not presented as representative samples of error rates.Source: §F.1 · p. 35, Figure A10 · p. 36, Figure A11 · p. 37

G RELEASED MATERIALS AND THEIR LIMITS

  • 99.35: released GPT Image 2 [Low] records reproduce the bilingual prompt-macro Composite, but do not establish other configurations’ coverage or score distributions.The records report automatic ratings, not correctness labels; broader comparisons require configuration details, image coverage, raw judge responses, and planned-row status.Source: G RELEASED MATERIALS AND THEIR LIMITS · p. 38

H HUMAN ALIGNMENT

  • Ten participants evaluated automatic scores, but no quantitative inter-rater or human–judge agreement is reported; a score of 100 is a rubric ceiling, not character accuracy.Exact reproduction requires transcription checks, and selected cases do not estimate scoring-error frequency or establish close-ranking reliability.Source: H HUMAN ALIGNMENT · p. 38
Loading 2610.09823v1…