Source-linked AI summary
Editing Text in the Wild
Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, Xiang Bai
TL;DR
Scene text editing seeks to replace words in natural images while preserving realistic text style and background texture. The paper proposes SRNet, an end-to-end network that separates text conversion, background inpainting, and fusion. It reports strong visual and quantitative results, including on ICDAR 2013, and demonstrates word-level editing, text erasure, and cross-language use.
Problem
Scene text editing must replace or modify words while preserving the diverse style of the text and the consistency of complex background regions.
Method
SRNet decomposes editing into skeleton-guided text conversion, background inpainting, and fusion modules that jointly generate the edited image.
Results
The method achieves strong subjective visual realism and objective quantitative scores on ICDAR13, with reported capability for word-level editing, text erasure, and cross-language editing.
Takeaways & Limitations
Modular decomposition supports realistic scene-text editing across intra-language editing, AR translation, information hiding, and other applications described by the paper.
Takeaways & Limitations
The authors identify failures on very complex structures, rare font shapes, residual shadows, complicated spatial layouts, and boundaries around text, attributing them to insufficiently represented training samples.
Abstract
from arXiv · showhide
In this paper, we are interested in editing text in natural images, which aims to replace or modify a word in the source image with another one while maintaining its realistic look. This task is challenging, as the styles of both background and text need to be preserved so that the edited image is visually indistinguishable from the source image. Specifically, we propose an end-to-end trainable style retention network (SRNet) that consists of three modules: text conversion module, background inpainting module and fusion module. The text conversion module changes the text content of the source image into the target text while keeping the original text style. The background inpainting module erases the original text, and fills the text region with appropriate texture. The fusion module combines the information from the two former modules, and generates the edited text images. To our knowledge, this work is the first attempt to edit text in natural images at the word level. Both visual effects and quantitative results on synthetic and real-world dataset (ICDAR 2013) fully confirm the importance and necessity of modular decomposition. We also conduct extensive experiments to validate the usefulness of our method in various real-world applications such as text image synthesis, augmented reality (AR) translation, information hiding, etc.
1 INTRODUCTION
Scene text editing replaces words in natural images while preserving realistic text and background styles. SRNet addresses diverse style-transfer and background-retention challenges through modular, end-to-end trainable components.
- Motivation: Scene text editing replaces text instances while retaining the styles of both text and background for realistic output.The task supports applications including text image synthesis, advertising photo editing, text correction, and augmented-reality translation.
- Challenges: Text style transfer must preserve language, font, color, orientation, stroke size, and spatial perspective across target text.Complex scenes also make background consistency difficult, especially when shorter target text leaves regions requiring texture completion.
- Approach: SRNet decomposes scene text editing into jointly trainable text conversion, background inpainting, and fusion modules.The modular design learns from image pairs and targets the task’s separate style-transfer and background-retention requirements.
- Approach: The text conversion module transfers source text style, the inpainting module removes original strokes and fills texture, and fusion combines foreground and background information.The conversion module includes skeleton-guided learning to preserve target-text semantics, while inpainting follows a bottom-up feature-fusion design.
- Contributions: SRNet is presented as the first end-to-end trainable word- or text-line-level scene text editor and is reported to outperform most image-to-image translation GANs.The paper also reports strong performance for intra-language editing, cross-language AR translation, and information hiding.
2 RELATED WORK
Prior work addressed character-level style transfer and scene-text erasure, but not arbitrary word- or text-line-level style migration with complete background retention. SRNet combines these capabilities in a more efficient word-level editing framework.
- Text Style Transfer: Earlier methods studied GAN-based image translation, style transfer, and character-level stylized text generation.These approaches commonly focused on transferring style between individual characters or generating stylized character images.
- Integrated Editing: SRNet’s overall design uses skeleton-guided text conversion, background inpainting, and fusion to combine transferred text with a completed background.The figure identifies these three modules as the network’s main structure.
- Word-Level Editing: SRNet targets arbitrary text styles and special effects at word or text-line level rather than character level.Word-level annotations are easier to obtain, editing is more efficient, and the editor can adapt foreground placement for words of different lengths.
- Text Erasure: Scene-text erasure methods remove text strokes and inpaint corresponding regions, but patch-based approaches can incur substantial computational cost.The cited related work motivates integrating text erasure with style transfer for realistic scene-text editing.
- Integrated Editing: The proposed framework integrates text-style transfer with complete background erasure or inpainting to produce more realistic edited images.This combination addresses the limitation of methods that transfer only character appearance while ignoring background texture consistency.
3 METHODOLOGY
SRNet decomposes scene text editing into text conversion, background inpainting, and fusion modules that jointly preserve target semantics, source text style, and background texture.
- Overview: SRNet takes source-style and target-text images and outputs a target skeleton, style-transferred foreground, inpainted background, and final edited image.The network is trained end to end on paired inputs and supervised outputs.
- Text Conversion Module: The text conversion module transfers font, color, geometric deformation, and other foreground-style attributes from the source image to the target text.It encodes source and target images, concatenates their features, and decodes a style-retained target-text image.
- Text Conversion Module: Skeleton-guided learning preserves target-text semantics by predicting a skeleton map and combining it with decoder features.The skeleton response block uses up-sampling layers and sigmoid activation, with Dice loss for skeleton reconstruction.
- Background Inpainting Module: The background inpainting module erases source text strokes and fills the region with appropriate texture using encoder-decoder processing and skip connections.The skip connections retain richer texture from matching encoder feature maps, while adversarial and L1 losses support realistic restoration.
- Fusion Module: The fusion module combines the generated foreground and background information to synthesize the edited scene-text image.Its objective includes adversarial, L1, perceptual, and style losses, with VGG-19 features used for perceptual and style comparisons.
- Training and Inference: Training uses synthetic paired images with similar style but different text, while inference crops target patches, edits them, and pastes them back into their original locations.Synthetic examples include style image, target image, foreground text, text skeleton, and background components.
4 EXPERIMENTS
The experiments evaluate SRNet through qualitative results, comparisons with other neural-network methods, and ablation studies on synthetic and real-world data.
- Experimental Setup: The experimental section presents qualitative scene-text editing results, comparisons with other neural-network methods, and an ablation study.These experiments are intended to assess the model's editing ability and the effectiveness of its approach.
- Datasets: Synthetic data pairs images with similar style but different text and provide background, foreground, and text-skeleton ground truth.The training set contains 50,000 images and the test set contains 500 images.
- Datasets: The real-world evaluation uses ICDAR 2013, which contains 229 training pictures and 233 test pictures of horizontal English scene text.Images include detailed text labels and horizontal-rectangle annotations for one or more text boxes.
4.2 Implementation Details
The evaluation uses standard image-generation metrics on synthetic data and recognition accuracy plus visual assessment on real data. Training uses Adam with specified normalization and learning-rate settings, while recognition is evaluated on rendered ICDAR 2013 word images.
- Implementation Details: The network is trained with Adam, β1 = 0.5, β2 = 0.999, spectral normalization, and batch normalization.The learning rate decays from 2 × 10^-4 to 2 × 10^-6 after 30 epochs.
- Evaluation Metrics: MSE, PSNR, and SSIM evaluate synthetic test images, where lower MSE or higher PSNR and SSIM indicate greater similarity to ground truth.These metrics are computed only on cropped regions because the network input is a cropped image.
- Evaluation Metrics: Real-data evaluation uses recognition accuracy and qualitative visual assessment because the real dataset lacks paired data.Recognition accuracy is computed on cropped regions.
- Evaluation Metrics: The attention-based recognizer is trained on Jaderberg-8M synthetic data and ICDAR 2013 training data with rotation and x-axis resize augmentation.Each model renders 1000 word images from ICDAR 2013 test data for evaluation.
4.4 Ablation Study
The ablation study shows that skeleton guidance, modular decomposition, and VGG loss each improve edited-image quality. Removing decomposition produces the weakest results, while removing other components harms structure, appearance, or both.
- Ablation Setup: The ablation study evaluates settings that remove skeleton guidance, decomposition, or VGG loss.Figure 5 presents qualitative examples of these component removals.
- Skeleton-guided Module: Removing the skeleton-guided module causes local text bending or breakage and worsens every metric, especially SSIM.The full model better preserves transferred text structure and deformation.
- Benefits from Decomposition: Removing the background-inpainting branch leaves original text remnants and vague text-background regions, with generally worst metrics.Decomposition helps learn clear strokes while reducing learning complexity.
- Discussion of VGG Loss: Removing VGG loss produces unrealistic appearance, inconsistent character scales, weaker text structure, and worse results on all metrics.The ablation compares perceptual and style-loss effects against the full model.
4.5 Comparison with Previous Work
SRNet is compared with pix2pix using matched training configurations. The comparison reports better foreground transfer, background retention, text regularity, and overall quantitative performance for SRNet.
- Experimental Setup: The comparison gives both methods the same training configurations while concatenating the style image and target text for pix2pix.This setup enables pix2pix to implement multiple-style translation.
- Qualitative Comparison: SRNet preserves foreground text transfer and background texture more correctly than pix2pix, with regular edited structure and consistent fonts.The background texture is described as more reasonable, and the result is similar to the real picture overall.
- Quantitative Comparison: SRNet is superior to pix2pix on all reported metrics.The paper uses pix2pix because no prior work focused on word-level text editing and adapts it for multiple-style translation.
4.6 Cross-Language Editing
The cross-language experiment tests English-to-Chinese editing on ICDAR 2013 to assess generalization. The generated Chinese text retains source-image color, geometric deformation, background texture, and input-text structure.
- Experimental Setup: The experiment evaluates English-to-Chinese text editing on the ICDAR 2013 test set for visual and augmented-reality translation applications.English fonts are manually mapped to common Chinese fonts using stroke similarity in size, thickness, and inclination.
- Results: Chinese outputs preserve color, geometric deformation, background texture, and character structure from the input text.The passage describes these outputs as realistic and as demonstrating the method’s synthesis performance.
4.7 Text Information Hiding
The background inpainting subtask supports word-level information hiding by selectively erasing text and filling its location with appropriate texture.
- Word-level text erasure can select specific text areas instead of removing all text from an image.This makes partial erasure more practical for information-hiding applications.
- The background subtask outputs an erased image as part of the text-editing system.
- Erased text locations are filled with appropriate texture in the resulting images.
4.8 Failure Cases
The method generally handles scene images but fails on complex text structures, rare fonts, residual shadows, and difficult boundaries.
- The method may fail on text with very complex structures or rare font shapes.
- Failure cases include residual shadows, suboptimal background erasure, and boundaries that are not transferred with the text.
- The authors attribute these failures to inadequate representation of such samples in the training data.
- The failure-case examples compare source images on the left with edited results on the right.
5 CONCLUSION AND FUTURE WORK
The paper presents an end-to-end, three-step network for word-level scene-text editing that preserves text style, with reported visual and quantitative success and additional erasure and cross-language capabilities. Future work targets more complex scenarios, broader language coverage, easier use, and improved evaluation.
- Conclusion: The network transfers foreground text style with skeleton guidance, erases the original text with appropriate texture, and merges the transferred text with the background.
- Conclusion: The paper reports outstanding visual realness and objective quantitative scores on the ICDAR13 dataset.
- Conclusion: The network also supports text erasure and cross-language editing, with effectiveness verified through comprehensive ablation studies.
- Future Work: Future work will address more complex scenarios, make the model easier to use, edit across more language pairs, and develop better evaluation metrics.