Source-linked AI summary
FlowTranslator: Training-Free In-Image Machine Translation with Flow-Based Editing
Zhixian Li, Xingsong Ye, Yongkun Du, Yunyan Xiao, Yanglin Huang, Pinxuan Chen, Jiyu Zhuang, Cong Xu, Zhineng Chen
TL;DR
IIMT must translate text embedded in images while preserving the surrounding image, but cross-lingual training pairs for rendering are scarce. FlowTranslator uses frozen, training-free flow editing with trajectory controls to address an ignition failure in small text regions. It reports strong results across benchmarks, including 40.1% exact-match accuracy on ScenePair photographs.
Problem
IIMT rendering remains a bottleneck, while cross-lingual paired images are scarce and synthetic training data narrows the fonts, layouts, and backgrounds editors encounter.
Source: §1 · pp. 1–2Method
FlowTranslator replaces trained rendering with region-masked flow editing on a frozen text-to-image model, using OCR, an LLM, and mask-confined trajectory controls.
Source: §1 · p. 3Results
FlowTranslator reports the highest average BLEU and COMET on PRIM and 40.1% exact-match accuracy on ScenePair, outperforming six released editing systems.
Source: §1 · p. 3Takeaways & Limitations
Results across synthetic and real imagery support FlowTranslator as a strong training-free baseline for IIMT.
Source: §1 · p. 3
Abstract
from arXiv · showhide
In-image machine translation (IIMT) replaces text in an image with its translation while preserving the surrounding visual context. Existing task-specific models achieve convincing performance but require massive paired training images before and after translation. As advanced text-to-image (T2I) models have reached a practically usable level of multilingual text generation, we argue that this capability can be transferred to IIMT and present FlowTranslator. It is a training-free IIMT framework that first employs an OCR model and an LLM to generate the translation prompt, which is injected into the T2I-based diffusion model. To optimize the accurate and natural generation of translated text, we introduce two trajectory controls: Region Coupling Release (RCR) partially decorrelates target-branch noise inside the text region early in the trajectory, while Coupling Expansion (CE) amplifies the emerging edit displacement mid-trajectory. Under the EasyOCR read-back protocol, FlowTranslator achieves average PRIM BLEU and COMET scores of 20.6 and 57.2, respectively, substantially outperforming the strong training-based baseline UniTranslator. On the more challenging and realistic whole-image ScenePair photographs translated into Chinese, it reaches 40.1\% exact-match accuracy and surpasses existing editing systems, some of which only address the text-rendering stage rather than the complete IIMT task. These results establish flow-based editing as a strong, fully training-free baseline for IIMT.
1 Introduction
FlowTranslator replaces the trained rendering stage of the IIMT cascade with frozen, training-free flow editing. Its mask-confined trajectory controls address an ignition failure in small text regions, and the system reports competitive results across IIMT benchmarks.
1 Introduction
- Cross-lingual training pairs are scarce outside a few high-resource directions, so training-based editors rely on synthetic text that narrows the fonts, layouts, and backgrounds seen in training.Earlier erase-and-paste systems can also leave seams, mismatched typography, and destroyed textures.Source: §1 · pp. 1–2
- Usable multilingual text generation by recent flow-matching models motivates testing whether a frozen text-to-image model can render IIMT translations without task-specific training.Prior training-free editors address a single word or crop with a user-provided target string, not automatic localization and translation of every text region.Source: §1 · p. 2
- FlowTranslator replaces the training-based rendering stage with region-masked flow editing on a frozen text-to-image model, while OCR and a frozen LLM locate and translate text.No parameter is updated or fine-tuned for IIMT; the diffusion prior generates translated text in the original image rather than pasting it.Source: §1 · pp. 2–3
- Coupled flow editing can fail to start when translated text occupies a small part of the image, because the source and target branches initially differ too little to provide an edit signal.RCR partially decorrelates target-branch noise early; CE amplifies the emerging displacement mid-trajectory, with neither adding forward passes or updating latents outside the text mask.Source: §1 · p. 3
- FlowTranslator reports the highest average BLEU and COMET on PRIM, highest average BLEU on Translatotron-V(ision), and 40.1% exact-match accuracy on ScenePair, outperforming six released editing systems.The evaluations cover three benchmarks, seven languages, and nine translation directions, without IIMT training data.Source: §1 · p. 3
2 Related Work
Prior IIMT work includes cascaded and end-to-end systems, while scene-text editing has evolved from separate glyph pasting to learned editors. FlowTranslator extends training-free flow editing to the full task of automatic localization and translation.
2 Related Work
- Published full-pipeline IIMT systems train or fine-tune at least one component on IIMT data, whereas FlowTranslator updates no parameters.Unified models have narrowed the gap and report competitive performance with cascaded baselines on established benchmarks.Source: §2 · p. 3
- Scene-text editors evolved from adaptive or stroke-level appearance transfer to diffusion editors, but cross-lingual training still depends on scarce paired images or synthetic composites.TextFlow demonstrates training-free editing on word crops; FlowTranslator extends it to full IIMT with automatic localization and translation.Source: §2 · p. 4
- FlowEdit constructs an inversion-free ODE between source and target text conditions, and FlowTranslator adds spatially masked RCR and CE controls for text-minority ignition.These controls fit a broader shift from constructing new ODEs toward controlling flow-editing trajectories.Source: §2 · p. 4, Figure 2 · p. 4
3 Method
FlowTranslator uses a frozen T2I flow model as a text-region renderer, with OCR and an LLM preparing the source and target conditions. Region Coupling Release and Coupling Expansion address the near-zero edit displacement that can arise when the branches share noise and differ only in a small text region.
3.1 Overall Framework
- FlowTranslator combines OCR, LLM translation, and region-masked flow editing on frozen models, generating translated text into the original image without training or fine-tuning.The detected text boxes are rasterized and dilated into a mask; latents outside it remain unchanged.Source: §3 · p. 4, §3.1 · p. 5
- Near-identical source and target branches can yield near-zero velocity differences and displacement, leaving source text unchanged; increasing the gain cannot escape this fixed point.The method addresses this by changing branch coupling rather than simply scaling the velocity difference.Source: §3.1 · p. 5
3.2 Region Coupling Release
- Region Coupling Release partially decorrelates target-branch noise inside the text mask early, allowing the branches to diverge while preserving their marginal noise distribution.Its scheduled release closes at later steps, restoring exact coupling for glyph refinement; full decorrelation could degrade the edit into mask-region regeneration.Source: §3.2 · pp. 5–6
3.3 Coupling Expansion
- Coupling Expansion feeds accumulated edit displacement back into the target branch, with a middle-trajectory gate that avoids early direction injection and late glyph distortion.The feedback is self-limiting within the edit window.Source: §3.3 · p. 6
- CE alone carries most of the ablation gain, while RCR adds a further increment and composes positively with CE across most translation directions.The controls are complementary: RCR opens room for ignition, and CE amplifies the emerging edit.Source: §3.3 · p. 6
4 Experiments
The experiments compare FlowTranslator across TTV, PRIM, and ScenePair, then examine its trajectory controls and edit-window setting. Results cover synthetic and photographed imagery, including whole-image English-to-Chinese translation.
4.1 Evaluation and Implementation Details
- The evaluation uses official benchmark splits and shared OCR boxes and target translations on ScenePair to isolate the rendering stage.PRIM and TTV text is read back with EasyOCR; the implementation uses Z-Image-Turbo with 50 solver steps and an edit window of 45.Source: §4.1 · pp. 6–7
4.2 Main Results
- 23.19 average BLEU on TTV exceeds UniTranslator’s 21.17, with FlowTranslator higher on three of four directions and lower on Fr→En.Its average SSIM is 0.746, below the 0.77–0.80 range of IIMT-tuned systems.Source: §4.2 · pp. 7–8, Table 1 · p. 7
- 20.6 average BLEU and 57.2 COMET on PRIM exceed UniTranslator’s 12.8 BLEU and 50.7 COMET, despite its fine-tuning on 7M in-domain pairs.FlowTranslator's controls raise its masked-FlowEdit baseline from 16.9 to 20.6 average BLEU and from 53.3 to 57.2 COMET.Source: §4.2 · p. 8, Table 2 · p. 9
4.3 Real-Scene Generalization
- 40.08% exact-match ACC on ScenePair, with FlowTranslator exceeding six released editing systems in whole-image English-to-Chinese translation.The control configuration raises exact-match ACC from 33.44% to 40.08%.Source: §4.3 · p. 8, Table 3 · p. 9
4.4 Ablations
- Adding RCR and CE raises average PRIM BLEU from 16.88 to 20.57, a relative gain of 22%.CE alone accounts for roughly 90% of the full 3.69-point improvement; RCR composes positively with CE on four of five directions.Source: §4.4 · p. 9, Table 4 · p. 9
- The edit-window setting peaks at n_max=45: shrinking it to 43 leaves visible source text, while extending it to 48 over-edits and damages layout.BLEU falls to 18.95 at 43 and 14.99 at 48; the experiments use n_max=45 for T=50.Source: §4.4 · pp. 9–10
- Control gains are largest on PRIM (+22% relative average BLEU) and smallest on TTV (+2% on Ro→En), with the same configuration used across benchmarks.The authors interpret this pattern as consistent with harder-to-ignite, small-glyph regimes benefiting most from trajectory intervention.Source: §4.4 · p. 10
5 Conclusion
FlowTranslator uses frozen flow-based editing with two trajectory controls as a training-free approach to IIMT. It reports leading benchmark scores, while inheriting its backbone’s script-coverage limits.
5 Conclusion
- FlowTranslator replaces the trained rendering stage of an IIMT cascade with region-masked editing on a frozen text-to-image model and adds two controls for cross-lingual, text-minority editing.The controls are Region Coupling Release and Coupling Expansion.Source: §5 · p. 10
- On PRIM, FlowTranslator attains the highest reported average BLEU and COMET, and on Translatotron-V(ision) it attains the highest average BLEU without IIMT training data.On ScenePair photographs translated into Chinese, it reaches 40.1% exact-match accuracy and exceeds six released editing systems.Source: §5 · p. 10
- The method inherits its frozen backbone’s script coverage; Cyrillic rendering failed in the authors’ experiments.Extending the controls to backbones with broader script support is left for future work.Source: §5 · p. 10
Appendix
The appendix diagnoses the two mask-confined trajectory controls using four matched experimental arms.
Appendix
- The four arms are Masked FlowEdit, +RCR, +CE, and +RCR+CE, allowing the controls to be examined separately and together.The figures denote Masked FlowEdit as FlowEdit.Source: Appendix · p. 14
- The diagnostics test whether RCR relaxes early branch coupling and whether CE amplifies accumulated displacement without destabilizing the latent trajectory.Source: Appendix · p. 14
- All diagnostics use 12 paired images under the default setting T=50, n_max=45, κ=0.5, and c=0.4.Source: Appendix · p. 14
A.1 Trajectory Diagnostics for Region Coupling Release
The paired trajectory diagnostics show that RCR strengthens the early in-mask edit signal, while the prescribed noise-correlation schedule is closely realized and RCR and CE activate in different trajectory phases.
A.1 Trajectory Diagnostics for Region Coupling Release
- The diagnostics compare four arms on matched trajectories using 12 images, 50 scheduler steps, and 45 edit steps per image.RCR's extra noise comes from a dedicated fixed-seed generator, preserving matched trajectories; probes require no extra network evaluations or random draws.Source: §A.1 · pp. 14–15
- RCR lowers early in-mask velocity alignment from 0.740 to 0.481 and raises edit-signal concentration from 1.94 to 2.63 (+36%).Both changes hold on 12/12 images; +CE alone shifts alignment by only −0.002.Source: Figure S1 · p. 14, §A.1 · p. 15
- The prescribed and measured noise correlations differ by a mean absolute error of 0.0032 across steps and images.RCR stops adding decorrelation at τlo = 0.3, restoring branch-noise coupling for late glyph refinement.Source: Figure S1 · p. 14, §A.1 · p. 15
- At the first step, RCR perturbation averages 0.962 across 12 images, while CE perturbation is 0.000.RCR is active at the noisy end; CE is inactive at zero displacement and peaks mid-trajectory under its sin²(πt) gate.Source: Figure S1 · p. 14, §A.1 · p. 15
A.2 Trajectory Diagnostics for Coupling Expansion
The displacement diagnostics associate CE with larger final in-mask edits and RCR with earlier edit onset; the combined controls produce little change in latent scale.
A.2 Trajectory Diagnostics for Coupling Expansion
- CE raises final in-mask displacement from 0.440 to 0.533 (+21%); with RCR, it reaches 0.513 (+16%).The increases occur on 11/12 and 10/12 images, respectively.Source: §A.2 · p. 16, Figure S2 · p. 16
- RCR advances the shared-threshold ignition step from 9.50 to 7.75, while +RCR+CE reaches 7.67.RCR alone is earlier on 9/12 images; the combined arm is earlier on 10/12.Source: §A.2 · p. 16, Figure S2 · p. 16
- With both controls, the latent maximum changes by 1.6%, from 3.991 for Masked FlowEdit to 4.053.CE perturbation peaks near t≈0.5 and returns nearly to zero before glyph refinement; out-of-mask displacement is zero at every logged step.Source: §A.2 · p. 16, Figure S2 · p. 16
- Figure S3 pairs source images with FlowTranslator outputs for full-image scene-text translation into Chinese.Each source image is shown on the left and its output on the right.Source: Figure S3 · p. 17
A.3 Visualization and Discussion
The qualitative examples show full-image scene-text translation into Chinese across varied source languages and text layouts, with the surrounding scene and local text appearance preserved.
A.3 Visualization and Discussion
- Across source languages and text layouts, FlowTranslator replaces selected text with Chinese while preserving the surrounding scene and local text-region appearance.The examples are consistent with stronger edits inside the detected text mask and unchanged background latents.Source: §A.3 · p. 17