Real In, Real Out: What If We Only Use Real Data for Scene Text Editing?

1 Introduction

• Prior STE methods rely heavily on synthetic data or disentanglement, which can leave source text, create misspelled or missing characters, or produce mismatched styles and artifacts.Inpainting approaches can also lack enough style cues to recover the desired appearance.Source: §1 · p. 2
• Shuffle&Crop performs best among the explored real-data schemes by disrupting text semantics while retaining style cues.The schemes include column-wise or block-wise shuffling, cropping, and masking.Source: §1 · p. 3
• TextRIRO achieves more realistic and accurate edits on real benchmarks, and reports 1.91% overall and 3.28% Multi-words accuracy gains for STR over a real-only baseline.The gains are reported on Union14M-Benchmark after mixing TextRIRO-3M with real data.Source: §1 · p. 3
• TextRIRO trains solely on real data with self-paired supervision, avoiding explicit or implicit disentanglement and mitigating synthetic-domain artifacts and hallucinations.It uses a conditional diffusion model to edit text from a real image.Source: §1 · p. 3
• TextRIRO-3M is a large-scale realistic composite dataset for long-text recognition, built using TextRIRO-edited and original images.Training recognition models with it significantly improves STR.Source: §1 · p. 3

Model Perspective on STE

• STE models progressed from GANs to diffusion models and unified multimodal models, with methods adding curved-text editing, character-level control, inpainting, OCR guidance, and segmentation-based generation.Examples include SRNet, SwapText, STEFANN, DiffSTE, DiffUTE, TextDiffuser, and UDiffText.Source: §2 · p. 3
• UMM approaches include multilingual editing through ControlNet, direct style-condition disentanglement, and joint editing, removal, and recognition frameworks.AnyText and AnyText2 use ControlNet; DARLING and RS-STE connect scene-text tasks.Source: §2 · p. 4

Data Perspective on STE

• Training data evolved from synthetic rendered pairs to real-data erasing, writing-back, and reverse self-supervision, while inpainting datasets expanded real-data scale.The related work notes that style consistency can be difficult when surrounding context lacks similar references.Source: §2 · p. 4

3 Method

• The ideal STE objective maps a source image and target text to a target image, but real training triplets are rarely available.The method therefore introduces self-paired training, a unified architecture, and STR dataset expansion.Source: §3 · p. 5

3.1 Self-Paired STE Data

• Shuffle&Crop disrupts semantic continuity while preserving local appearance: it shuffles equal bins along the text's long axis, then crops, or crops directly with probability 1 − p_shuffle.Shuffling alone can fragment glyphs; cropping preserves local tilts and curvature in complex layouts, and the combination improves robustness and diversity.Source: §3.1 · p. 5

3.2 Model Architecture

• TextRIRO conditions a diffusion generator on multi-scale style features and character-level text features, using a ViT style encoder and an STR-pretrained text encoder trained on real scene text.The VAE and text encoder are frozen while the style encoder and U-Net are jointly trained; the design is backbone-agnostic.Source: §3.2 · p. 6

3.3 Generating Long-Text Recognition Data

• For long-text recognition data, TextRIRO edits each real input into same-length different content, filters edits whose STR prediction matches the target, then concatenates accepted edits with originals.The concatenation doubles sequence length and broadens coverage of rare long and multi-word instances.Source: §3.3 · p. 6

4.1 Editing Performance Comparison

• TextRIRO approaches natural-image realism while improving editing correctness; RS-STE can exceed the recognizer’s ground-truth accuracy but has poorer FID.Qualitative comparisons attribute residual artifacts and style loss to competing methods’ editing strategies and synthetic-data reliance.Source: §4.1 · p. 7, Table 1 · p. 7
• 47.64% variable-length editing accuracy on RIROCommon, with equal-length accuracy 19.07% higher than RS-STE, shows TextRIRO outperforms alternatives on both tasks.The comparison covers equal-length substitutions and variable-length targets without length limits.Source: Table 2 · p. 7, §4.1 · p. 8
• TextRIRO reaches 86.67% average accuracy, improving 9.78% over the next-best approach and widening the gap to 11.69% on irregular samples.The train-on-edits STR proxy compares edited data from different STE methods.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 Ablation Study

• Shuffle&Crop gives the best accuracy–realism trade-off; STR pre-training, SD 2.1 VAE reconstruction, and rectangular 64×256 inputs also improve the design.Without paired-data perturbation, the model degenerates to identity copying; Mask destroys both content and style, while Block introduces distribution shift.Source: §4.2 · p. 8, Figure 4 · p. 9
• Shuffling contributes most for straight text, while cropping contributes most for curved and multi-oriented text by preserving local tilt and curvature.The attribution study compares Shuffle only, Crop only, and Shuffle&Crop on IC13, CUTE, and SVTP.Source: §4.2 · p. 9, Table 4 · p. 9
• ControlNet-based TextRIRO outperforms its DiT-based conditional-control variant, whose compact token-level conditioning is too coarse for fine scene-text cues.Multi-scale feature injection better preserves local style cues across spatial resolutions.Source: §4.2 · p. 9, Table 5 · p. 10
• Across three recognizers, RS-STE exceeds the real-image upper bound in accuracy but has worse FID, supporting the interpretation that it produces less realistic, recognizer-friendly images.The reported FID values are 49.59 for RS-STE and 41.53 for the real-image upper bound.Source: §4.2 · p. 10, Table 6 · p. 10

4.3 Synthetic Data Utility

• On Union14M-Benchmark Multi-Words, TextRIRO data improves recognition by 2.19%, and scaling to 3M raises MAERec Multi-Words accuracy to 90.17%.Real-Concat also helps, but less than TextRIRO-generated data; the authors attribute the difference to style inconsistency between independently concatenated images.Source: Table 7 · p. 10, §4.3 · p. 11, Table 8 · p. 11

5 Conclusion

• TextRIRO learns editing from real data by disrupting semantic content with shuffle and crop operations while maintaining style, avoiding synthetic triplets and disentanglement modules.The diffusion-based model achieves near-realistic fidelity and robust content accuracy across multiple benchmarks.Source: §5 · p. 11
• TextRIRO surpasses existing STE methods in both correctness and visual realism across multiple benchmarks.Source: §5 · p. 11
• Training STR models on TextRIRO-3M yields significant gains, while accurate and realistic multilingual STE in low-resource settings remains future work.Source: §5 · p. 11

• The illustration contrasts paired synthetic-data training with self-paired real-data training.Source: Figure 6 · p. 14, Appendix · p. 14

6.1 Model Formulation Details

• The style encoder extracts ViT features, uses adaptive average pooling and a zero convolution for each pyramid level k, then aggregates features from all levels.Target text is truncated or padded to maximal character length LM, and each character is mapped to a vocabulary index.Source: §6.1 · p. 14

6.2 Model specific settings

• On ScenePair, pshuffle = 0.7 gives the best overall accuracy and NED while maintaining competitive FID, so it is the default setting.Lower values reduce accuracy, while higher values slightly degrade FID without substantial gains on other metrics.Source: §6.2 · p. 15, Table 9 · p. 15

6.3 Datasets and Implementation Details

• Training uses 3.22M labeled U14M-Filter instances, while inference edits unperturbed real images using a new target string.Training perturbs each real source and reconstructs its original label; the unavoidable train–inference mismatch is addressed with Shuffle&Crop, and an unperturbed-source ablation reaches ACC 4.37.Source: §6.3 · pp. 15–16
• At CFG scale ω = 2.2, TextRIRO balances ACC of 85.23%, NED of 0.964, and FID of 41.53.Increasing ω generally improves text correctness but worsens FID beyond 1.5; ACC peaks at 5.0 while NED continues to improve.Source: §6.3 · p. 16, Table 11 · p. 17

6.4 Additional Editing Evaluation

• TextRIRO achieves the best LPIPS and FID, while RS-STE leads SSIM and PSNR and TextRIRO ranks second on both pixel-level metrics.The authors note that ScenePair references are not strictly pixel-aligned, so realistic outputs may score lower on SSIM and PSNR despite better perceptual or distributional fidelity.Source: §6.4 · p. 17, Table 12 · p. 17

6.5 Additional TextRIRO Details

• Asymmetric concatenation balances odd- and even-length gains at +2.48 and +2.25, while adding synthetic data raises ACC and NED but worsens FID.The asymmetric recipe mixes 0.5M asymmetric and 0.5M symmetric samples; the quality filter retains highly reliable samples but can reject visually correct examples in unusual styles.Source: §6.5 · pp. 18–19, Table 13 · p. 18, Table 15 · p. 18

7.1 Character-Aware Embeddings

• STR-pretrained embeddings place samples with similar character compositions or lengths close together while separating distinct samples.‘1’ and ‘l’ have similar glyphs, but case variants remain distinct; this separation prevents case confusion and enables accurate character-level generation.Source: §7.1 · p. 19

7.2 TextRIRO-3M Data View

• TextRIRO-3M doubles original text length while preserving style, but its symmetric construction produces only even lengths.Asymmetric augmentation removes this length artifact when broader coverage is needed.Source: §7.2 · p. 19, Figure 8 · p. 20, Figure 9 · p. 21

7.3 Enhanced STR Visualization

• TextRIRO-3M training produces more accurate and robust MAERec predictions, especially for long, complex sequences.The authors report reduced errors in difficult cases.Source: §7.3 · p. 20, Figure 10 · p. 21

7.4 Bad Cases and Discussion

• Long-text handling is constrained by an STR-pretrained encoder and TextRIRO training data centered on short text.The proposed response is to edit shorter fragments individually and concatenate them.Source: §7.4 · p. 20
• Extreme curvature and large geometric deformation remain difficult because regular layouts dominate training.The paper reports equal-/variable-length accuracy of 64.87/48.12 on CUTE and 79.69/42.31 on SVTP, outperforming compared baselines.Source: §7.4 · p. 20
• Sparse symbol coverage and Latin-heavy real labeled datasets constrain special-symbol synthesis and cross-lingual generalization.The authors say balanced real labeled data with broader character and language coverage is needed for stable multilingual generation.Source: §7.4 · p. 20
• Shuffle preserves stroke thickness, while Crop supplies scale variation for variable-length editing; together, they are more robust than either alone.Pure-Shuffle examples remain in training with non-trivial probability, anchoring stroke-width preservation.Source: §7.4 · p. 21

Broader impacts and safeguards

• The release is intended to include research-use and acceptable-use notices, STR-based quality filtering, and encouragement of provenance tracking or watermarking at deployment.The authors note possible misuse for falsifying text in images alongside benign applications.Source: §8 · p. 21

Existing assets and licenses

• Released artifacts are to follow the original licenses and terms for the listed public datasets, model components, and baselines.The paper identifies TextRIRO-3M as a newly constructed derived dataset and documents its sources and construction.Source: §8 · pp. 21–22

Human assessment protocol

• Three assessors judged generated image-label pairs correct only when every visible character matched the target label in order.The protocol involved visual inspection only; no personal data, crowdsourcing platform, or external paid workers were used.Source: §8 · p. 22