Source-linked AI summary
UM-Text: A Unified Multimodal Model for Image Understanding and Visual Text Editing
Lichen Ma, Xiaolong Fu, Gaojing Zhou, Zipeng Guo, Ting Zhu, Yichun Liu, Yu Shi, Jason Li, Junshi Huang
TL;DR
Visual text editing requires understanding instructions and reference images while maintaining stylistic consistency, but prior approaches often require manually specified text attributes. UM-Text uses VLM-based planning, multimodal condition fusion, regional glyph supervision, and staged training, and reports state-of-the-art performance across public benchmarks.
Problem
Visual text editing requires interpreting instructions and reference images while generating style-consistent text, whereas prior methods often manually specify text content and attributes.
Method
UM-Text combines VLM-based planning, UM-Encoder multimodal condition aggregation, regional consistency losses, three-stage training, and the UM-DATA-200K dataset.
Results
UM-Text reports state-of-the-art performance across multiple public benchmarks, consistently outperforming competing approaches on the reported metrics except for one lower LPIPS score than DreamText.
Takeaways & Limitations
A unified multimodal model can support diverse visual text generation and editing applications from natural-language instructions while adapting layout and attributes to context.
Abstract
from arXiv · showhide
With the rapid advancement of image generation, visual text editing using natural language instructions has received increasing attention. The main challenge of this task is to fully understand the instruction and reference image, and thus generate visual text that is style-consistent with the image. Previous methods often involve complex steps of specifying the text content and attributes, such as font size, color, and layout, without considering the stylistic consistency with the reference image. To address this, we propose UM-Text, a unified multimodal model for context understanding and visual text editing by natural language instructions. Specifically, we introduce a Visual Language Model (VLM) to process the instruction and reference image, so that the text content and layout can be elaborately designed according to the context information. To generate an accurate and harmonious visual text image, we further propose the UM-Encoder to combine the embeddings of various condition information, where the combination is automatically configured by VLM according to the input instruction. During training, we propose a regional consistency loss to offer more effective supervision for glyph generation on both latent and RGB space, and design a tailored three-stage training strategy to further enhance model performance. In addition, we contribute the UM-DATA-200K, a large-scale visual text image dataset on diverse scenes for model training. Extensive qualitative and quantitative results on multiple public benchmarks demonstrate that our method achieves state-of-the-art performance.
1 Introduction
UM-Text enables visual text editing through natural-language instructions by using multimodal context to adaptively generate text content, layout, and implicit attributes. It combines VLM-based planning, condition aggregation, regional supervision, and a 200k-image dataset for unified generation and editing.
- UM-Text automatically generates compelling text content, appropriate layout, and visually harmonious text images with implicit text attributes.
- The framework integrates multimodal understanding into visual text generation and editing, supporting four generation and editing patterns through reference-image and instruction context.
- UM-Text produces results for image editing, image translation, and poster design, with text bounding boxes adaptively generated by the model.
- UM-Encoder aggregates T5, character-level visual, and context embeddings as conditions for visual text generation and editing.
- Regional consistency losses supervise glyph generation in both latent and RGB spaces, while a three-stage training strategy further supports the framework.
- UM-DATA-200K provides 200k diverse image pairs with and without visual text for VLM pre-training.
2 Related Work
Related work improves visual text rendering with diffusion models, text or glyph embeddings, and language-model-based layout prediction, but limitations remain for contextual visual text editing. UM-Text addresses these limitations by combining character-level glyph information with multimodal context and jointly predicting text and layout.
- Diffusion-based visual text generation seeks accurate text images whose visual elements and text layout are harmoniously integrated.
- Character-aware models outperform character-blind counterparts, while other methods align language-model text features with visual features or directly incorporate glyph images.
- UM-Encoder integrates T5 embeddings, character-level visual embeddings, and VLM embeddings to improve text accuracy and stylistic consistency with the reference image.
- LLM-based layout methods learn positions from text modality and cannot be directly applied to visual text editing, whereas UM-Designer predicts layouts and text related to the reference image.
3 Methodology
UM-Text combines multimodal context understanding, unified condition encoding, regional consistency supervision, and progressive training for natural-language visual text generation and editing. Its framework targets accurate, context-aware layouts and text with improved glyph fidelity and visual harmony.
- Framework: UM-Text integrates a VLM, UM-Encoder, diffusion transformer, regional consistency losses, and a three-stage training strategy for visual text generation and editing.The VLM predicts text content, layout, and implicit attributes from instructions and reference images, while the UM-Encoder aggregates multiple conditions.
- UM-Encoder: UM-Encoder combines instruction, reference-image, character-level glyph, and implicit attribute embeddings into conditions for the diffusion model.UM-Designer predicts text content, layout, and attributes, while character-level rendering and OCR provide fine-grained glyph information.
- Regional Consistency Loss: Regional Consistency Loss supervises visual-text structure in latent and RGB spaces using localized regions and edge maps.The latent-space loss operates on velocity fields, while the RGB-space loss compares Canny edges in localized regions.
- Training Strategy: The three training stages pretrain UM-Designer, pretrain the diffusion model, and align UM-Encoder with the diffusion model.The stages progressively establish image-text understanding, text-generation capability, and connections between condition representations and diffusion generation.
- Results: UM-Text significantly improves high-fidelity, harmonious visual text generation, consistently outperforming competing methods across AnyText-benchmark metrics.The reported evaluation covers OCR accuracy and realism metrics, while qualitative comparisons emphasize accurate, coherent, and background-consistent text.
4 Experiments
UM-Text is evaluated across public benchmarks, qualitative comparisons, multi-turn editing, and ablations. It consistently improves text accuracy and style consistency, while its advantages extend to complex multilingual and multi-line scenarios.
- Quantitative Results: UM-Text consistently outperforms competing approaches on the AnyText benchmark for Chinese and English across OCR accuracy and realism metrics.The evaluated metrics are Sen.ACC, NED, FID, and LPIPS.
- Quantitative Results: UM-Text surpasses previous approaches in SeqAcc and FID on UDiffText, although DreamText achieves a higher LPIPS score.The authors attribute the lower LPIPS to colors and textures that better match image style during text reconstruction.
- Quantitative Results: UM-Text significantly outperforms previous approaches on Sen.ACC and NED for both Chinese and English on the UMT-benchmark.All compared methods use layout and text produced by UM-Designer.
- Qualitative Results: Qualitative comparisons show accurate, coherent, and visually harmonious multilingual text that blends with backgrounds, whereas competing methods exhibit blurred, duplicated, or incorrect glyphs.The comparison covers English and Chinese multi-line text scenarios against AnyText, AnyText2, and FLUX-Text.
- Qualitative Results: In multi-turn editing, UM-Text maintains glyph integrity and background consistency in complex multi-line scenarios, while ChatGPT-4o often introduces unnecessary text modifications.The comparison covers poster design, image editing, and image translation.
- Ablation Study: Adding the character-level visual encoder and VLM embedding improves baseline text generation, while LRCL and LRCI yield improvements of 4.8% and 4.2%, respectively.The ablation uses FLUX-Fill as the baseline and is conducted on a subset of AnyWord-3M.
5 Conclusion
UM-Text unifies multimodal understanding and image editing for complex visual text tasks through natural language instructions. Its VLM, UM-Designer, UM-Encoder, regional consistency loss, training strategy, datasets, and benchmarks support accurate, style-consistent text generation and editing.
- Conclusion: UM-Text combines VLM and diffusion models with three-stage training and UM-Designer for flexible visual text generation and editing through natural language instructions.The framework predicts text layout and content while integrating multimodal conditions for downstream generation.
- Conclusion: UM-Encoder fuses VLM, character-level visual, and T5 embeddings to improve understanding of scene images and text glyphs.Regional consistency loss provides supervision for fine-grained visual text glyph information.
- Conclusion: UM-DATA-200K and the UMT-benchmark provide resources for training and evaluating instruction-based visual text editing.The dataset contains diverse layouts and texts, while the benchmark evaluates the complete pipeline.
- Conclusion: Extensive qualitative and quantitative results demonstrate the superiority of the proposed approach.