VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

1 INTRODUCTION

• Existing benchmarks primarily assess visual quality and related properties, while text-focused benchmarks cover limited longer passages and one relies entirely on labor-intensive human evaluation.Scalable evaluation of longer scene text remains underexplored.Source: §1 · p. 2
• Across 11 state-of-the-art models, the lowest overall WER is 0.250; the Keyframe-Guided framework reduces Minimax H3 WER by 32.5% and improves Video Score.The full framework also improves both aggregate metrics over direct first-frame conditioning.Source: §1 · p. 2
• VTR-Bench provides 300 prompts across five application scenarios, with multiple text targets, reference strings, and carrier annotations in each scene.Its automated evaluation separates carrier-specific text transcription and WER from a 20-question prompt-specific assessment of scene and motion requirements, with both branches validated against human alignments.Source: §1 · pp. 2–3
• The Keyframe-Guided Agentic Framework coordinates image and video generation with visual evaluation, iterative refinement, and candidate selection.A Director agent guides this process through visual feedback.Source: §1 · pp. 2–3

2.1 VISUAL TEXT RENDERING IN VIDEOS

• Video text-rendering approaches include text preservation, synthetic-data training, glyph animation, and editing, while VTR-Bench evaluates prescribed text on carriers alongside scene requirements.The related work distinguishes text rendered into existing videos from approaches that animate glyphs or edit source-video text.Source: §2.1 · p. 3

2.2 VIDEO GENERATION BENCHMARKS

• Video-generation benchmarks cover general quality and prompt adherence, specialized capabilities such as compositionality and physical commonsense, and audiovisual generation.Dedicated text benchmarks include human evaluation of on-screen text and broader audiovisual evaluation that incorporates scene-text rendering.Source: §2.2 · p. 3
• VTR-Bench’s overview connects expert-designed scenario categories and human-reviewed prompt construction to separate Video Score and carrier-specific WER evaluation.Video Score measures scene and video requirements with a 20-question checklist; WER compares carrier-specific transcriptions with reference text.Source: Figure 2 · p. 4

3.1 DATASET CONSTRUCTION

• VTR-Bench builds prompts from human-curated scene seeds, then validates text and carrier feasibility with iterative VLM feedback and final human review.Three reference images are checked per candidate; after three consecutive unsuccessful checks, the candidate is reconstructed before validation restarts.Source: §3.1 · p. 4

3.2 DATASET STATISTICS

• The 300 prompts cover five scenarios and 1,202 text blocks, testing both multiple targets per scene and reproduction of longer passages.Text blocks range from 1 to 275 tokens, with a median of 23; required text per video ranges from 58 to 496 tokens, with a median of 102.5.Source: §3.2 · pp. 4–5

3.3 BENCHMARK EVALUATION

• Evaluation separates scene and motion fulfillment, measured by prompt-specific 20-question Video Score, from carrier-specific transcription compared with reference text using WER.A shared tokenizer preserves case, content-bearing symbols, and individual CJK characters; bounded WER uses α = 5 by default.Source: §3.3 · p. 5

3.4 KEYFRAME-GUIDED AGENTIC GENERATION

• The Director agent uses visual feedback to refine or select first frames, then revises motion plans and selects videos based on sampled-frame review.The video generator receives the selected first frame, original prompt, and motion plan while retaining the same model weights as direct video generation.Source: Figure 4 · p. 6, §3.4 · p. 6

4.1 EXPERIMENT SETTINGS

• Both open-source and proprietary models were evaluated; proprietary generation used 720p, videos were fixed at 10 seconds and 24 FPS, and Qwen3.8-27B evaluated videos and text.Open-source models used vllm-omni with their default optimal settings except for video length.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• 0.250 is the lowest overall WER recorded among the evaluated models, while text fidelity varies substantially within and across model groups.Minimax H3 stands out among open-source models at WER 0.447; Wan3.0 leads text fidelity among proprietary models across scenarios.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 FAILURE ANALYSIS

• Higher resolution reduces Minimax H3’s WER at both tested durations, but LTX-2.3 remains near WER 1 across configurations and shorter duration gives no consistent improvement.The comparison tested high/low resolution and 10/5-second durations for both models.Source: Figure 5 · p. 7, §4.3 · p. 8
• Across 13,213 text targets, 23.31% were judged unreadable versus 9.34% with carriers not found, indicating difficulty recovering text even when its carrier is located.Figure 6 reports proportions across three evaluator states: carrier not found, text unreadable, and transcribed.Source: §4.3 · p. 8, Figure 6 · p. 8
• Transcription coverage does not ensure accurate content: Kling v3.0 transcribes 79.62% of targets but has WER 0.979, while Minimax H3 has 85.61% transcription and WER 0.447.The analysis identifies both legibility on intended carriers and faithful reproduction of prescribed content as priorities.Source: §4.3 · p. 8

4.4 EFFECT OF TEST-TIME REFINEMENT

• 0.4468 to 0.3015: the agentic framework reduces overall WER by 32.5% relative to direct generation while raising Video Score from 0.7562 to 0.8315.Both metrics improve across all five scenarios, using the same video generator.Source: §4.4 · p. 8, Figure 7 · p. 9
• The full framework reduces overall WER by 9.4% relative to I2V and achieves the highest Video Score in every scenario, though I2V has the lowest WER in UI scenes.The framework’s WER advantage over I2V spans four of five scenarios and is largest in scientific videos.Source: §4.4 · p. 9

4.5 HUMAN ALIGNMENT

• 92.15% of automated scene-and-motion judgments agree with human majority votes, and video-level text evaluation reaches Lin’s concordance coefficient 0.9835 with mean absolute error 0.0407.The human comparison covers 100 sampled videos, 2,000 binary queries, and 404 text blocks.Source: §4.5 · p. 9

5 CONCLUSION

• The 11-model evaluation finds visual text rendering remains challenging across open-source and proprietary systems, with the strongest model recording overall WER 0.250.Failures include unreadable text and legible text that conveys incorrect content; the agentic framework reduces overall WER by 32.5% while improving Video Score on Minimax H3.Source: §5 · pp. 9–10

A MORE RELATED WORK

• Image text-rendering research has moved from explicit character and layout controls toward native text modeling and targeted refinement, emphasizing completeness, placement, dense content, and character accuracy.The reviewed work also studies product posters and article-level infographics.Source: A MORE RELATED WORK · p. 14

B.1 SCENE-GROUNDED PROMPT CONSTRUCTION

• Category-specific templates turn human-screened scene seeds into prompts linking exact text, its carriers, and semantic role to the depicted activity across five application scenarios.Construction requirements preserve the scene and purpose, keep text spatially coherent and internally consistent, and assign one group the semantic-anchor role.Source: §B.1 · pp. 14–15

B.2 VISUAL FEASIBILITY VALIDATION

• Visual feasibility validation jointly reviews three Qwen-Image reference images and revises prompts only for recurring structural problems attributable to the prompt, followed by final human review.Spelling errors, malformed glyphs, inaccurate wording, and isolated failures are not grounds for revision.Source: §B.2 · pp. 16–17

• The analyses use main-experiment videos and the Section 3.3 evaluation protocol, with α = 5 except in the parameter sweep.Source: C ADDITIONAL EVALUATION RESULTS · p. 18

C.1 RESULTS WITH AN ALTERNATIVE EVALUATOR

• Both evaluators give identical overall rankings across 11 models for WER and Video Score, with Spearman correlation 1.000 for each metric.Table 3 reports VTR-Bench results evaluated by Qwen3.6-27B.Source: §C.1 · p. 18, Table 3 · p. 18
• Wan3.0, Minimax H3, and Seedance2.5 occupy the top three positions in text fidelity.Source: §C.1 · p. 18
• Qwen3.6-27B assigns higher Video Scores throughout, while the largest difference in overall WER is 0.017, indicating score calibration changes without changing model ordering.Source: §C.1 · p. 18

C.2 HUMAN ALIGNMENTS AND EVALUATOR SELECTION

• Qwen3.8-27B agrees with human CoQ judgments on 92.15% of queries, versus 91.60% for Qwen3.6-27B, and performs better on all four text-block alignment metrics.Source: §C.2 · p. 18
• At video level, Qwen3.8-27B has higher Pearson correlation and Lin’s CCC, while Qwen3.6-27B has higher Spearman correlation and lower MAE.Source: §C.2 · p. 18
• Qwen3.8-27B is selected for the main evaluation based on stronger agreement on individual queries and text blocks, using the same 100 annotated videos for the comparison.Source: §C.2 · p. 18

C.3 SENSITIVITY TO THE WER TRUNCATION PARAMETER

• The sensitivity analysis recomputes WER from saved transcriptions while varying how many tokens beyond the reference length are retained, for α ∈ {0, . . . , 10}.Source: §C.3 · p. 18
• Table 5 marks α = 5 as the default setting and indicates the lowest WER in each column for each evaluator.Source: Table 5 · p. 19
• Wan3.0, Minimax H3, and Seedance2.5 retain the three lowest overall WERs, in that order, across every α setting under both evaluators.Table 5 reports overall WER for α = 0–10, with α = 5 marked as the default.Source: §C.3 · p. 19, Table 5 · p. 19

C.4 VIDEO EVALUATION BY DIMENSION

• Each dimension score is the proportion of satisfied queries assigned to that dimension, pooled across scenarios; prompts without applicable queries do not contribute.Scene Attributes includes text-carrier presence and integration, while Entity Presence covers subjects and objects; the All column pools every query.Source: §C.4 · pp. 19–20
• Across 11 models, Scene Attributes, Entity Presence, and Spatial Relationship score higher than Motion Adherence and Temporal Consistency; Wan3.0 leads the first three dimensions.HappyHorse1.1 has the highest Motion Adherence and shares the highest Temporal Consistency with Seedance2.5.Source: §C.4 · p. 20, Table 6 · p. 20
• Kling v3.0 and Seedance2.5 both score approximately 0.79 overall, but Kling leads Spatial Relationship at 0.8118 versus 0.7503, while Seedance leads Temporal Consistency at 0.7049 versus 0.6361.The passage says the aggregate score brings these distinct profiles close together.Source: §C.4 · p. 20

C.5 EXTENDED FAILURE ANALYSIS

• Diagnostics pooled across 13,213 targets include 6,433 descriptions for unreadable targets and 6,549 accompanying transcriptions.Evaluators describe observable difficulties when a target is entirely unreadable or unreadable spans are omitted; Figure 8 counts terms in the selected diagnostic vocabulary.Source: §C.5 · p. 20, Figure 8 · p. 21
• Small and blurry lead unreadability diagnostics, with 3,595 and 3,196 occurrences, while handwriting, viewing distance, and distortion also recur.A carrier may be recognizable while its writing lacks enough detail for transcription.Source: §C.5 · pp. 20–21, Figure 8 · p. 21
• Partially leads recoverable-text descriptions at 1,645 occurrences, while misspelled and garbled appear 602 and 480 times.Recoverable text may remain partly obscured or blurry, and transcription does not ensure well-formed or correct words; WER measures correspondence to requested content.Source: §C.5 · p. 21
• Generated scenes retain recognizable layouts even when text becomes pseudo-words, malformed characters, distorted handwriting, or unreadable detail.The examples span structured interfaces, Chinese text, handwriting, and printed text.Source: §C.5 · p. 21
• β = 8/3 becomes β = 4/3; other cases omit requested timing and temperature or render prompt instructions as storefront text.These errors concern required textual content even when the writing appears legible.Source: §C.5 · p. 21, Figure 15 · p. 24, Figure 16 · p. 24