PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

Ziying Zhang, Litao Li, Junchao Liao, Tianyi Zeng, Siyu Zhu, Long Qin, Zhenghao Zhang

arXiv:2609.36598v3cs.CV

TL;DR

Visual text is fragile in video, yet existing evaluations largely treat it as incidental or assess it in static images. VidScribe benchmarks four video text regimes with a 12-axis dataset and gated metrics, finding that performance varies by task and that text recognition can coexist with faulty glyphs. Its diagnostics also support preference optimization that improves visual text fidelity and temporal preservation.

  • Problem

    Existing video benchmarks largely treat text as incidental, while static image-text evaluations miss whether characters remain legible, stable, and editable over time.

    Source: §1 · p. 2

  • Method

    VidScribe evaluates four video text regimes using 803 human-verified samples across 12 factor axes and a track-grounded, gated suite of 11 shared metrics and 2 task-specific probes.

    Source: §1 · p. 2, §3.2.1 · p. 5

  • Results

    Video text capability is non-monolithic: content recognition can coexist with faulty glyphs, I2V is more reliable than V2V editing, and text-centric structural and temporal factors produce greater degradation than adverse imaging conditions.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Benchmark-derived preference pairs measurably improve visual text fidelity and temporal preservation through preference optimization.

    Source: §1 · p. 3

Abstract

from arXiv · show

A video can exhibit convincing motion and photorealism yet fail immediately when visual text collapses. Unlike generic scene content, visual text is unforgiving in video generation: minor stroke corruption, temporal instability, or editing errors instantly break legibility and realism. Existing benchmarks overlook this challenge by treating text as incidental or using static OCR metrics that ignore temporal dynamics. We introduce VidScribe, a unified diagnostic benchmark spanning four generation regimes: writing from language (T2V), transferring text identity from a reference (R2V), sustaining text under dynamics (I2V), and localized text editing (V2V). VidScribe contains 803 human-verified samples across a 12-axis conditionally orthogonal factor space covering Intrinsic Text Properties, Physical Imaging Conditions, and Temporal Behavior. For reliable evaluation, we build a track-grounded, gated suite with 11 shared metrics and 2 task-specific probes under strict measurability conditions. Benchmarking 11 commercial and open-source systems shows that video text capability is non-monolithic, with content recognition decoupled from stroke-level glyph correctness. Performance is highly task-asymmetric: I2V sustains text most reliably, whereas V2V editing is the primary bottleneck. Counter-intuitively, degradation concentrates on a small subset of text-centric structural and temporal factors rather than adverse imaging conditions. Further probes show that visual references improve glyph and typographic fidelity rather than content accuracy, while localized editing fails to isolate target text without corrupting undeclared source text. Beyond evaluation, VidScribe also provides an actionable training signal, where benchmark-aligned preference optimization measurably improves visual text generation. https://huggingface.co/datasets/Vicky0720/VidScribe.

1 INTRODUCTION

Visual text remains poorly evaluated in video despite its sensitivity to small rendering errors and temporal instability. VidScribe benchmarks text writing, transfer, persistence, and editing, revealing distinct capabilities and stronger failures in localized editing than in text persistence.

Motivation

  • Existing video benchmarks treat text as incidental, while image-text benchmarks rely mainly on static OCR or single-image editing and miss dynamic-video failure modes.These include temporal character decay, carrier detachment, and incomplete erasure during editing.Source: §1 · p. 2

Benchmark contribution

  • VidScribe evaluates T2V, R2V, I2V, and V2V with 803 human-verified samples across 12 factor axes and a gated suite of 11 shared metrics plus 2 task-specific probes.The probes target localized editing residue and preservation of source text, while shared metrics cover text fidelity, temporal stability, typographic instruction compliance, and non-text preservation.Source: §1 · p. 2

Benchmark findings

  • Models recognize text content more reliably than they preserve glyph structure, sustain text most reliably in I2V, and struggle most with V2V editing.Reference-guided transfer outperforms text-only writing, while degradation concentrates on content evolution, foreground occlusion, and layout hierarchy rather than adverse imaging conditions.Source: §1 · p. 3

Diagnostic probes

  • Visual exemplars improve glyph structure and typographic style without materially improving spelling, while V2V edits can leave residual artifacts or corrupt undeclared text.The results suggest text preservation and appearance transfer are more reliable than localized rewriting under spatio-temporal constraints.Source: §1 · p. 3

Training signal

  • Benchmark diagnostics converted into verifiable preference pairs produce measurable gains in visual text fidelity and temporal preservation through preference optimization.The optimization is performed on an open-source video generator.Source: §1 · p. 3

2 RELATED WORK

Prior visual-text work progressed from compositing and GAN pipelines to diffusion methods with structural controls, but dynamic video text remains under-characterized. Existing video and image-text benchmarks leave this gap because they focus on broad video quality or static text evaluation.

2.1 VISUAL TEXT GENERATION AND EDITING

  • Diffusion methods improved 2D text generation using structural priors, but visual-text generation and editing in dynamic video remain largely unaddressed.Earlier compositing and GAN approaches often had blurry glyphs and boundary artifacts; later work expanded control over typography, stroke styles, multilingual rendering, and localized erasure.Source: §2.1 · p. 3

2.2 BENCHMARKING GENERATIVE VIDEO AND VISUAL TEXT

  • Video benchmarks emphasize realism, motion, or prompt adherence, while image-text benchmarks assess static accuracy and style, leaving temporal decay and localized editing residue unmeasured.Figure 2 organizes VidScribe’s data distributions across regimes, domains, styles, factor axes, spatial layouts, and linguistic properties.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3 METHOD

VidScribe combines a four-regime, 12-axis dataset with track-grounded evaluation, gated evidence pathways, and an explicit mapping from factor axes to metrics. The design supports controlled diagnosis of text rendering, temporal behavior, instruction compliance, and localized editing.

3.1.1 DATA CONSTRUCTION AND TASK ROUTING

  • VidScribe uses 803 human-verified samples across 12 axes and four regimes: writing from prompts, transferring reference text, sustaining text under dynamics, and localized video editing.An automated feasibility filter removes physically contradictory factor combinations before instantiation.Source: §3.1.1 · p. 4

3.1.2 DATA DISTRIBUTION AND TAXONOMIC ANALYSIS

  • Samples span five V2V edit operations, nine real-world domains, multiple styles and materials, calibrated spatial layouts, and English and Chinese text.The dataset includes non-semantic sequences to decouple glyph rendering from language priors, and its factor baselines support comparisons between Regular and Hard configurations.Source: §3.1.2 · p. 5

3.2.1 EVALUATION METRICS

  • Evaluation associates region-restricted OCR detections across frames and applies measurability gates before scoring 11 shared metrics and 2 task-specific probes.Shared metrics cover text fidelity, temporal stability, typographic instruction compliance, and background preservation; the V2V probes detect obsolete text inside edit regions and check text outside them.Source: §3.2.1 · p. 5

3.2.2 HIERARCHICAL HYBRID EVALUATION FRAMEWORK

  • Three evidence pathways route metrics to deterministic expert scoring, hybrid geometric evidence with VLM arbitration, or rubric-grounded VLM verification.The routes address low-level physical and perceptual signals, context-dependent spatial attributes, and semantic fidelity or discrete state transitions, respectively.Source: §3.2.2 · p. 6, Figure 3 · p. 6

3.3 FACTOR–METRIC MAPPING

  • The factor–metric design map assigns each axis a primary evaluation metric through definitional or mechanism-based assignments.For example, Content Evolution maps to Content Stability, while Illumination and Text-Background Separability map to Color Alignment because their described degradation mechanisms affect apparent color.Source: §3.3 · p. 6

4 EXPERIMENTS

VidScribe benchmarks visual text generation across multiple video tasks and factor conditions. Results show substantial task and factor asymmetries, while diagnostic probes identify distinct strengths and failures in reference-guided generation, editing, and preference optimization.

4.1 MAIN RESULTS

  • A1 reached 82.7%, but A2 remained below 60.0%; Wan 3.0 led A1 at 82.7% while Seedance 2.5 led A2 at 58.2%.MiniMax H3 had the highest Overall score (79.2%) but led no individual metric, underscoring that lexical, structural, and typographic skills diverge.Source: Table 1 · p. 7, §4.1 · p. 7
  • 82.2% for I2V versus 65.6% for V2V—a 16.6% gap—makes text preservation the strongest regime and localized editing the primary bottleneck.R2V outperformed T2V for 6 of 7 systems, with task means of 77.8% and 76.3%, respectively; Gemini Omni 1.1 Flash was the exception.Source: Table 2 · p. 7, §4.1 · pp. 7–8

4.2 FURTHER DIAGNOSTIC ANALYSIS

  • V2V editing can leave obsolete character fragments or alter adjacent valid text; low residue alone does not ensure source-text preservation.PixVerse V6 and Wan 3.0 had similar residue (18.6% vs. 19.5%), but source-text preservation was 33.1% vs. 69.5%.Source: Table 2 · p. 7, §4.2 · p. 8, Figure 5 · p. 9
  • Matched R2V–T2V comparisons found no spelling-accuracy gain from a visual reference, but A2 rose +13.4% (54.3% to 67.7%).Stroke Weight Agreement gained +10.8%, and Typographic Style Matching gained +9.5%.Source: Figure 4 · p. 7, §4.2 · p. 8, Figure 5 · p. 9

4.1 MAIN RESULTS

  • Content Evolution, Foreground Occlusion, and Layout Hierarchy reduced Overall by 7.9%, 5.0%, and 4.1%, respectively, while imaging-factor drops were smaller.Separability and Illumination reduced Overall by 1.6% and 0.8%; structural and temporal factors degraded multiple metric groups together.Source: Table 3 · p. 8, §4.1 · p. 8

4.3 HUMAN PREFERENCE ALIGNMENT AND EVALUATION ROBUSTNESS

  • A2 and C3 correlations with expert consensus were 0.84 and 0.91; both Temporal Stability metrics exceeded 0.80, and D1 reached 0.96.Using Gemini-3.8-Flash instead of Qwen3.7-Plus produced closely matched correlations.Source: Table 4 · p. 9, §4.3 · p. 9

4.2 FURTHER DIAGNOSTIC ANALYSIS

  • 48.4% after DPO, up from 45.9% (+2.5%), on held-out VidScribe T2V; gains included Layout Hierarchy (+9.3%) and Glyph Correctness (+5.0%).Spatial Placement Accuracy increased +4.6%, and Content Accuracy increased +4.2%.Source: §4.2 · pp. 8–9

5 CONCLUSION

VidScribe unifies evaluation of visual text across four video-generation regimes with a 12-axis factor space and track-grounded metrics. Experiments identify decoupled skills and task asymmetries, and the authors present the benchmark as a resource for future progress.

5 CONCLUSION

  • VidScribe benchmarks visual text in T2V, R2V, I2V, and V2V using a 12-axis factor space and track-grounded evaluation.The benchmark covers writing, transfer, sustaining, and localized editing under physical-imaging and temporal conditions.Source: §5 · p. 9
  • Systems often maintain recognizable text content while failing on glyph correctness, and perform more reliably in I2V than V2V editing.Degradation is greater under text-centric structural and temporal dynamics than adverse imaging conditions.Source: §5 · p. 10
  • Visual references mainly improve glyph and typographic fidelity, while localized editing remains limited by incomplete source-text removal and weak preservation of undeclared text.Source: §5 · p. 10

AI USE STATEMENT

The authors report using generative AI for prompt generation and cleaning, proof-writing assistance, and translation.

AI USE STATEMENT

  • Generative AI tools were used to generate prompts and clean the prompt dataset, which underwent human review.Source: AI USE STATEMENT · p. 10
  • The authors also report AI assistance with writing proofs and translation.Source: AI USE STATEMENT · p. 10
  • They report no AI use for methodology design, implementation, qualitative analysis, or interpreting results, and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

VidScribe’s prompts and media were reviewed or screened, and the materials are intended solely for benchmarking.

ETHICS STATEMENT

A DATASET DETAILS

This section details VidScribe’s 803-sample, four-task benchmark, its conditionally orthogonal 12-axis design, task-specific construction and quality-control procedures, and track-grounded evaluation metrics. It also explains how background consistency and two editing probes are computed and averaged across applicable generated videos for each system.

A.1 SHARED METADATA SCHEMA AND SAMPLE RECORDS

  • VidScribe contains 803 human-verified, nominally five-second single-shot samples with shared metadata for task, 12 factor axes, ground truth, prompts, and optional visual inputs.Most samples are 1280 × 720 landscape; V2V also includes 720×1280 portrait samples. V2V records distinguish source and edited target text, while temporal switching and cropping have task-specific ground-truth fields.Source: §A.1 · p. 14

A.2 TASK-SPECIFIC CONSTRUCTION AND PROMPTS

  • The four tasks use distinct construction protocols: T2V specifications are checked and human-reviewed, R2V pairs text prompts with rendered references, I2V annotates source images, and V2V edits screened source videos.R2V uses 139 samples and references that provide text appearance but not scene layout or temporal behavior; I2V temporal prompts follow fixed source-side metadata, while V2V annotations pair source and target text with localized edit type.Source: §A.2 · pp. 15, 17

A.3 CONDITIONALLY ORTHOGONAL FACTOR SPACE

  • The 12-axis factor space captures distinct variation sources while excluding combinations that are physically undefined, internally inconsistent, or not reliably measurable.Feasibility checks include clip-level constraints, rendering scale, metric-facing measurability, font support, and task-specific checks, supporting controlled factor-based comparisons.Source: §A.3 · pp. 17–18

A.4 QUALITY CONTROL

  • Quality control combines deterministic schema and feasibility checks, expert review, and generated-output validation, with additional source-side screening for I2V and V2V.I2V targets must be character-level readable and localizable at 720p; V2V clips are screened for resolution, duration, text visibility, and single-shot continuity.Source: §A.4 · pp. 18–19

B.1 METRIC COMPUTATION DETAILS

  • Metric computation begins with block-level text and bounding-box recognition, then associates detections into tracks and matches each ground-truth block to at most one track.Content and appearance comparisons use fully visible instances; content accuracy aggregates consensus readings across observation segments, while glyph correctness flags defective glyphs in sampled frames.Source: §B.1 · pp. 19–20
  • Content stability compares one-second-window readings with a time-weighted modal anchor, while appearance stability uses flow-aligned LPIPS on adjacent visible-frame pairs.Appearance stability is applied only with at least five valid pairs and a valid-pair ratio of at least 0.5.Source: §B.1 · p. 20
  • Additional metrics assess color, stroke-weight, typographic-style, carrier-type, and spatial-placement agreement against declared properties or positions.Spatial placement is evaluated in the first half of the video on a 3 × 3 frame partition; multi-level layouts use a macro average over targets.Source: §B.1 · pp. 20–21
  • Layout hierarchy fidelity scores whether the primary target matches its declared hierarchy using three votes.The observed options are single level, two levels, three or more levels, and table-like.Source: §B.1 · p. 21
  • Background consistency averages pairwise cosine similarity across retained frames for T2V, R2V, and I2V; V2V measures preservation between corresponding source and output backgrounds.Text regions are masked, and frames whose text area nearly covers the image are discarded; V2V uses normalized background features from temporally corresponding frames.Source: §B.1 · p. 21
  • Editing residue counts obsolete source-text matches inside the edit region only when they persist for at least two consecutive frames.The probe uses a per-frame indicator for whether the edit-region transcription matches unmatched old-side content.Source: §B.1 · p. 21
  • Source text preservation is evaluated in one-second windows, with character alignment used to compute lower and upper bounds and the score averaging lower bounds across source lines.Source: §B.1 · p. 21
  • Scores are computed per generated video and then averaged across applicable videos for each system.Source: §B.1 · p. 21

B.2 CROSS-BACKBONE EVALUATOR ANALYSIS

The appendix tests evaluator robustness and human alignment, validates metric components, and examines metric coupling and factor sensitivity. Results show stable system comparisons across evaluators and concentrated degradation among select factors.

B.2 CROSS-BACKBONE EVALUATOR ANALYSIS

  • System profiles remain closely aligned across Qwen3.7-Plus and Gemini-3.8-Flash, although Gemini assigns lower scores on some metrics.Every system shifts by at most one rank, and the highest- and lowest-ranked systems remain unchanged in Overall score.Source: §B.2 · p. 22, Figure 7 · p. 22

B.3 HUMAN ALIGNMENT OF OVERALL SYSTEM RANKINGS

  • Only the top two systems switch positions between VidScribe and human Overall rankings; their Human Overall scores differ by 0.3%.Source: Table 7 · p. 23, §B.3 · p. 23

B.4 METRIC-LEVEL HUMAN EVALUATION PROTOCOL

  • Eight experts independently rated 950 videos sampled across 11 systems, with each video rated by at least five experts using metric-specific rubrics.Majority voting over the five independent responses supplied the human consensus for correlation analysis.Source: §B.4 · pp. 23–24

B.5 DEDICATED DISCRIMINATORS VERIFICATION

  • Dedicated discriminators reach 93.88% Top-1 accuracy on style and 94.67% on weight, with worst-class accuracies of 88.64% and 91.52%, respectively.They outperform both alternatives under both criteria; Qwen3.7-Plus reaches 14.15% accuracy on the weakest style class.Source: Figure 9 · p. 24, §B.5 · p. 24

B.6 METRIC SANITY CHECK ON IDEAL SYNTHETIC VIDEOS

  • On 500 ideal synthetic videos, text-fidelity metrics reached 100%, temporal-stability metrics reached 100% and 99.7%, and Overall reached 97.9%.Across all submetrics, scores ranged from 90.5% to 100%.Source: §B.6 · p. 24, Table 8 · p. 25

C.1 VIDEO-LEVEL A1–A2 CORRELATION ANALYSIS

  • The video-level correlation between Content Accuracy (A1) and Glyph Correctness (A2) is weak (ρ = 0.246).Videos with high lexical accuracy often do not achieve equally high glyph correctness.Source: §C.1 · p. 25

C.2 FACTOR SENSITIVITY RANKING

  • Seven of eight top-ranked factors retain significantly negative adjusted coefficients; F6 is the exception (pFDR = 0.095).The four tail factors—F9, F8, F3, and F7—do not retain significant coefficients after adjustment for task and the other factor indicators.Source: §C.2 · pp. 25–26, Table 9 · p. 26

C.2.1 FACTOR–METRIC CORRESPONDENCE

Assigned factor–metric relationships mostly align with their intended metrics, but several factors also produce larger effects on neighboring metrics. Significant effects are concentrated in temporal-behavior factors, with some effects extending to non-text preservation.

C.2.1 FACTOR–METRIC CORRESPONDENCE

  • All 12 assigned factor–metric cells are significant; the strongest mappings are F2 to C6 (−0.51), F5 to C2 (−0.41), F4 to C3 (−0.37), and F7 to C4 (−0.28).F12 has the largest mechanism-assignment effect on B1 (−0.40), while F10 on C5 is weakest (−0.10).Source: §C.2.1 · p. 26
  • F1 depresses C5 (−0.31) more than its assigned A1 (−0.20), while F11 and F10 have their strongest effects on unassigned D1 (−0.35 and −0.26).Across all cells, 58 are significant: 49 negative and 9 positive, with the clearest concentration among temporal-behavior factors.Source: §C.2.1 · pp. 26–27
  • F3 is the only factor with a significantly positive assigned-metric effect, on A2 Glyph Correctness (+0.21); F8 and F9 share C1 evidence by construction.F1’s stronger C5 effect is consistent with the added spatial burden of dense text layouts.Source: §C.2.1 · p. 27

C.3 BENCHMARK-ALIGNED PREFERENCE OPTIMIZATION PERFORMANCE

The authors turn VidScribe scores into preference pairs to fine-tune Wan2.2-TI2V-5B with benchmark-aligned DPO. The resulting model improves benchmark scores and is preferred in a user study across all four evaluated dimensions.

Preference data

  • VidScribe scores select the highest-Overall and lowest-Overall candidates from eight samples per prompt, producing 1,000 fixed preference pairs from 8,000 videos.The prompts are disjoint from the benchmark evaluation prompts.Source: §C.3 · p. 27

Objective and training

  • The training objective combines a pairwise preference term with a winner-anchor regularizer.Training uses transformer LoRA with rank/α = 64/64, a frozen base model, and 121-frame videos at 480 × 832 resolution.Source: §C.3 · p. 27

Automatic evaluation

  • 48.42% Overall, up from 45.92% (+2.50%), on 250 held-out T2V prompts after benchmark-aligned DPO.The largest gains are C6 (+9.33%), A2 (+5.00%), C5 (+4.62%), A1 (+4.17%), C4 (+4.00%), and B1 (+3.87%).Source: §C.3 · p. 27

User study

  • The DPO variant is preferred on all four human-evaluation dimensions: Text Fidelity 68.8%, Temporal Stability 54.8%, Instruction Compliance 54.0%, and Non-Text Preservation 51.6%.The user study uses the same 250 prompts.Source: §C.3 · p. 27

Conclusion

  • Together, automatic and human evaluations support VidScribe as a practical preference signal for improving video text generation.Qualitative comparisons are shown in Figure 11.Source: §C.3 · p. 27

C.4 TASK-WISE PERFORMANCE

Task-wise results show distinct model strengths, with V2V the weakest regime; separately, quality declines as samples accumulate more Hard factors, without evidence of a distinct threshold collapse.

C.4 TASK-WISE PERFORMANCE

  • 84.2% is MiniMax H3’s I2V Overall score, while V2V is weakest, led by Wan 3.0 at 70.3%.The I2V leaders Kling3.0-Omni and Seedance 2.5 each score 83.5%; V2V next-best scores are Gemini Omni 1.1 Flash at 69.3% and MiniMax H3 at 68.7%.Source: §C.4 · p. 28

C.5 CUMULATIVE DIFFICULTY AND QUALITY DEGRADATION

  • 80.1% at N = 0 falls to 70.6% at N = 8 as the number of Hard factors per sample increases.Higher-count bins are concentrated in T2V and R2V, while mid-range bins contain all four tasks.Source: §C.5 · p. 28

C.4 TASK-WISE PERFORMANCE

  • 80.9% gives Gemini Omni 1.1 Flash the highest T2V Overall score, followed by Seedance 2.5 and MiniMax H3 at 80.2%.Metric leadership varies: Gemini leads B1 and B2, while Wan 3.0 leads A1 and several instruction-compliance metrics.Source: §C.4 · p. 28, Table 12 · p. 29
  • 81.9% and 84.2% are MiniMax H3’s Overall scores in R2V and I2V, respectively; in both tasks, other models lead some individual metrics.Seedance 2.5 and Wan 3.0 follow MiniMax in R2V at 80.9% and 80.4%; Kling3.0-Omni and Seedance 2.5 each score 83.5% in I2V.Source: §C.4 · p. 28, Table 13 · p. 29

C.5 CUMULATIVE DIFFICULTY AND QUALITY DEGRADATION

  • No tested threshold from k = 2 to 9 is significant after correction; the smallest adjusted value is pFDR = 0.32 at k = 9, with positive rather than negative γ.The results support gradual quality degradation without evidence of a distinct collapse threshold.Source: §C.5 · p. 29

C.6 FAILURE MODE

Failures recur across all four generation regimes, from glyph and layout errors to temporal disruption and poorly localized edits. These patterns persist even among overall task leaders.

T2V

  • T2V failures include wrong rare glyphs, hallucinated or dropped tokens, and illegible multi-line signs.All four models substitute the leading rare glyph while roughly preserving P25; on the archive sign, outputs range from extra codes to faint fragments.Source: §C.6 · p. 29, Figure 13 · p. 32

R2V

  • R2V transfers glyph identity more reliably than weight, contrast, or layout; examples include text dissolving into a low-contrast wall and stacked codes flattening into a line.Only PixVerse V6 keeps the low-contrast poster line stable; HappyHorse preserves the stacked-code layout but renders it in a light weight.Source: §C.6 · pp. 29–30, Figure 14 · p. 33

I2V

  • I2V text can still break under mid-clip content changes or persistent occlusion, with models scrambling chart values or degrading and losing characters.A provided first frame does not prevent these failures.Source: §C.6 · p. 30, Figure 15 · p. 34

V2V

  • V2V edits fail through partial application or mis-localization: models alter adjacent code during targeted replacement or change content during a style-only recolor.For the style-only edit, HappyHorse changes content and PixVerse V6 and MiniMax H3 recolor only part of the entries; targeted replacement also corrupts surrounding code or leaves the line unchanged.Source: §C.6 · p. 30, Figure 16 · p. 35

Cross-task pattern

  • These recurring failure modes mirror the benchmark’s damaging axes and persist even for the overall leaders in the evaluated tasks.The reported modes include rare-character errors, dense-sign token loss, ignored reference styling, low contrast, scene-change or occlusion failures, and mislocalized edits.Source: §C.6 · p. 30

Related papers

Loading 2609.36598v3…