VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation

• Web image translation must recognize and translate embedded text while preserving visual-scene context, despite varied fonts, layouts, and backgrounds.This makes it more challenging than standard neural machine translation [16] [30].Source: §1 · p. 1
• Cascaded OCR-and-translation systems risk error propagation, whereas specialized end-to-end models often lack the scale and world knowledge needed for Web linguistic diversity.Source: §1 · p. 1
• Off-the-shelf LVLM encoders often prioritize high-level image semantics over character-level detail, while simple visual-feature concatenation fails to fuse details with multilingual context.These limitations can cause hallucinations or omissions in translation.Source: §1 · pp. 1–2
• Across eight translation tasks in three benchmarks, VaaWIT significantly outperforms open-source SOTA baselines and achieves competitive performance with proprietary models.It is comparable to GPT4.1 and Gemini2.5 Pro and surpasses them on several tasks.Source: §1 · p. 2
• VaaWIT’s DSAM uses semantic and detail streams with bidirectional cross-attention to fuse global context and character morphology.Semantic context guides detail recognition, and detail features inform semantic interpretation.Source: §1 · p. 2
• Its lightweight VAA dynamically modulates the frozen LLM’s internal representations with fused visual cues, avoiding the cost of full-parameter tuning.The visual cues ground generation in visual evidence.Source: §1 · p. 2

2 Preliminaries

• The model learns a multimodal mapping from raw Web images to target translation sequences by maximizing target-token log-likelihood autoregressively.The input is an image with embedded text, and the target is a translation sequence of length L.Source: §2 · p. 2
• Unlike standard machine translation from source text, the end-to-end setting must implicitly perform OCR and translation simultaneously from pixel-level input.Source: §2 · p. 2
• Complementary visual backbones provide high-level semantic features and fine-grained morphological and layout details as inputs to the Dual-Stream Attention Module.The semantic encoder can lose high-frequency spatial information, while the detail encoder captures fine-grained structures and layout.Source: §2 · p. 2

3.1 Framework Overview

• VaaWIT combines dual-stream visual encoding, feature fusion, and visual-aware LLM adaptation in an end-to-end pipeline bridging fine-grained perception and multilingual semantic reasoning.Source: §3.1 · p. 3, §3 · p. 3

3.2 Dual-Stream Attention Module (DSAM)

• DSAM projects semantic and detail features into a shared space, uses reciprocal cross-attention to refine each stream, then fuses them with an MLP into an LLM-dimension representation.Semantic queries retrieve relevant morphological details, while detail features enrich semantic representations; the fused sequence captures linguistic context and visual details for character recognition.Source: §3.2 · p. 3

3.3 Visual-Aware Adapter (VAA)

• VAA pools fused patches into a global context, gates a bottleneck adapter at each frozen-LLM layer, and adds its output residually, updating only adapter and gate weights.The gate selectively enhances or suppresses visual adaptation according to visual-signal confidence while the residual preserves pretrained linguistic knowledge.Source: §3.3 · pp. 3–4

3.4 Training

• Training freezes visual encoders and the LLM, first aligning DSAM/VAA by reconstructing image text, then jointly learning ITM, text-only TTL, and image-conditioned ITL with λ_ITL > λ_ITM > λ_TTL.Source: §3.4 · p. 4

4.1 Experimental Setup

• Experiments cover eight tasks across MIT-10M, ECOIT, and OPUS-MIT-5M, using BLEU and COMET to assess surface similarity and semantic fidelity.The task set spans English and Italian alongside Chinese, Japanese, Korean, Thai, and Hindi; comparisons include cascaded pipelines, zero-shot LVLMs, tuning strategies, and end-to-end translation models.Source: §4.1 · p. 4

4.2 Main Results

• On ZH-EN, VaaWIT surpasses EasyOCR with Google Translate and PP-OCR with Microsoft Translator cascades by more than 50 BLEU points; it also substantially outperforms zero-shot LLaMA3.2 (90B).Source: §4.2 · pp. 4–5
• On IT-EN, VaaWIT reaches 66.0 BLEU and 88.2 COMET, beating Full FT by 7.6 BLEU and 2.9 COMET with about 50M trainable parameters.Visual encoders and the LLM backbone remain frozen; only DSAM and VAA are trained, and the two-stage training takes about 18 hours.Source: §4.2 · p. 5
• VaaWIT performs consistently with both Qwen3 and LLaMA3.1 backbones, supporting the framework’s transferability across LLM architectures.The evaluation covers eight tasks spanning ZH-EN, EN-IT, EN-JA, IT-EN, JA-EN, HI-EN, KO-EN, and TH-EN.Source: §4.2 · pp. 4–5

4.3 Comparison with Image Translation Models

• On ZH-EN, VaaWIT achieves 65.9 BLEU and 94.8 COMET, exceeding Translatotron-V [16] by 13.3 BLEU and 11.7 COMET; on EN-IT it reaches 38.1 BLEU and 85.9 COMET.The comparison evaluates end-to-end image translation methods on ZH-EN and EN-IT.Source: §4.1 · p. 4, §4.3 · p. 5, Table 3 · p. 6

4.4 Ablation Study

• On ZH-EN, removing DSAM, VAA, or both lowers BLEU by 2.2, 0.9, or 4.6 and COMET by 5.4, 5.0, or 6.8, respectively, indicating synergistic contributions.Without DSAM, naive feature fusion misses complementary semantic and orthographic cues; without VAA, the LLM cannot dynamically regulate visual influence, especially for ambiguous or noisy images.Source: §4.4 · p. 6

5.1 Gating Strategy Analysis

• Global gating stays within 0.4 BLEU of the best configuration while using nearly half its resources, whereas layer-plus-token gating costs 90% more parameters and 56% higher latency.Layer-specific gating adds 40% more parameters for minimal improvement, while token-dependent gating raises inference latency by 42% with almost no performance gain.Source: §5.1 · pp. 6–7

5.2 Vision Feature Fusion Strategies

• DSAM improves on Simple Concat by nearly 3 BLEU on both ZH-EN and HI-EN, while explicit bidirectional cross-attention outperforms one-way and self-attention fusion.On ZH-EN, One-way CA (Sem→Det) scores 64.5 BLEU/91.0 COMET and One-way CA (Det→Sem) scores 64.2 BLEU/90.7 COMET, each 1.4–1.7 BLEU and 3.8–4.1 COMET below DSAM; on HI-EN, they score 39.2 and 38.9 BLEU versus DSAM’s 41.3. DSAM uses semantic context to refine visual features and visual cues to enrich semantics.Source: §5.2 · p. 7

5.3 VAA Insertion Strategies

• With controlled parameter counts, the Early Layers strategy has the lowest performance, reducing BLEU by 2.1 and 2.3 points on JA-EN and HI-EN, respectively, versus Uniform Insertion.Late Layers performs better than Early Layers but still lags significantly behind Uniform Insertion; findings suggest effective adaptation requires adjustments throughout the LLM’s processing depth.Source: §5.3 · p. 7

5.4 The Vision Encoder

• The mSigLIP–DINOv2 encoder pair performs best overall, exceeding the mSigLIP–MAE alternative by +2.1 BLEU / +4.0 COMET on ZH-EN.Dual-encoder variants outperform single-encoder baselines, and other pairs also improve over single encoders, supporting DSAM’s ability to integrate different visual encoders.Source: §5.4 · p. 8

5.5 Sensitivity Analysis of 𝜆in Stage 2

• On ZH-EN, performance remains relatively robust to tested Stage 2 loss-weight combinations, with only minor BLEU and COMET differences.The adopted combination is (0.3, 0.2, 0.5), and the analysis reports no significant fluctuations after full training.Source: §5.5 · p. 8, Table 8 · p. 8

5.6 Training Schedule Sensitivity Analysis

• Skipping Stage 1 reduces ZH-EN scores by 2.8 BLEU and 3.4 COMET (65.9→63.1 / 94.8→91.4), and doubling Stage 2 epochs without it still yields only 64.0 BLEU.Extending either stage to two epochs adds only +0.3–0.5 BLEU and +0.2–0.3 COMET; the 1+1 schedule achieves near-optimal performance with approximately 18 hours of training on 8×NVIDIA H20 GPUs. Stage 1 supplies a visual-language alignment initialization that extra task-specific training alone does not replace.Source: §5.6 · p. 8

5.7 Case Study

• Across two case studies, VaaWIT provided complete translation of scattered e-commerce image text and, for six IT-EN labels, translated UMIDO as Organic, whereas GPT4.1 and LLaMA3.2 (90B) omitted content.Source: §5.7 · p. 9

6 Related work

• Web image translation covers multilingual text in advertising, product displays, information dissemination, and user-generated content, where varied fonts, colors, and multilines impede global information access.Source: §6 · p. 9
• Cascaded OCR–MT pipelines suffer error propagation, while end-to-end models unify recognition and translation but remain constrained by task-specific data scale and diversity.End-to-end approaches have used multi-task learning, knowledge distillation, modality alignment, and multimodal representation learning.Source: §6 · p. 9
• Standard LVLMs face a visual representation gap and unresolved fusion-and-adaptation challenges; simple concatenation or gating fails to deeply combine multilingual semantics with fine visual detail.Mainstream visual encoders optimize image-level semantic understanding, whereas Web image translation requires fine-grained visual features.Source: §6 · p. 9
• VaaWIT uses DSAM bidirectional cross-attention to bridge the visual representation gap and VAA visual-aware dynamic gating for parameter-efficient, context-aware LLM adaptation.Source: §6 · p. 9

7 Conclusion

• VaaWIT combines DSAM for deep interaction between complementary visual features with VAA for parameter-efficient, dynamic adaptation within the LLM backbone.The framework is designed for multilingual web image translation.Source: §7 · p. 9
• Across several public benchmarks, VaaWIT outperformed previous methods and achieved performance comparable to commercial systems.The paper also reports parameter and training efficiency.Source: §7 · p. 9
• Future work could extend VaaWIT with layout-aware translation.Source: §7 · p. 9

A Implementation Details

• Both mSigLIP and DINOv2 visual encoders were kept frozen during training.Source: A Implementation Details · p. 11
• Experiments ran on NVIDIA H20 GPUs with CUDA 12.2.Source: A Implementation Details · p. 11
• The deep learning stack used torch 2.3.1, torchvision 0.18.1, and Transformers 4.57.0.Source: A Implementation Details · p. 11
• The implementation used CentOS release 7.5 and Python 3.9.12.Source: A Implementation Details · p. 11

A.1 Datasets

• Experiments used three public Web image translation datasets, with their statistics shown in Table 10.Source: §A.1 · p. 11

A.2 Two-Stage Training Settings

• Stage 2 mixes continued alignment, pure text translation, and end-to-end image translation data, with loss weights λITM=0.3, λTTL=0.2, and λITL=0.5.It uses 30% of Stage 1 data for continued alignment, source-target text pairs for translation, and complete image/source/target examples for image translation.Source: §A.2 · p. 11
• Stage 1 uses all MIT-10M, ECOIT, and OPUS-MIT-5M training data to align fused visual representations with textual semantics in the LLM embedding space.The DSAM and VAA modules are trained with autoregressive language modeling loss using image content and source-language text.Source: §A.2 · p. 11
• Table 12 covers performance degradation under different noise conditions on ZH-EN, with degradation percentages relative to baseline.The supplied passage gives the table scope but no cell values, so it does not establish which noise condition causes greater degradation.Source: Table 12 · p. 11
• Only DSAM and VAA, approximately 50M parameters, are updated; visual encoders and the LLM backbone remain frozen.Training uses one 8-GPU NVIDIA H20 node with DeepSpeed ZeRO Stage 2; Stage 1 takes about 6 hours and Stage 2 about 12 hours, totaling about 18 hours.Source: §A.2 · p. 11

A.3 Baselines and Fairness of Comparison

• All methods use the same test sets, metrics, and decoding settings; zero-shot LVLMs receive a unified translation prompt and use original-resolution images with greedy decoding and OCR/tools disabled.Commercial model evaluations for GPT-4.1 and Gemini 2.5 Pro were conducted in October 2025.Source: §A.3 · p. 11
• LoRA and full fine-tuning are applied to Qwen3-VL (8B) using VaaWIT’s Stage 2 data, while CoT uses prompting only; E2E baselines are reproduced or evaluated with their stated pipelines.The E2E models are retrained on the combined training dataset using original hyperparameters, except training-free AnyTrans, which uses its published pipeline.Source: §A.3 · p. 11

B.1 Robustness Analysis

• The study targeted practical web-image degradation, including compression artifacts, low-resolution captures, motion blur, and partial occlusions from watermarks or overlays.Robustness was tested on the ZH-EN task using the MIT-10M test set.Source: §B.1 · p. 11
• The evaluation applied Gaussian blur, JPEG compression at quality 30, 50% downsampling, 15% random-area occlusion, and mixed noise to all test images.Mixed noise combined all four degradations to represent co-occurring real-world web-image damage; VaaWIT was compared with Full Fine-Tuning.Source: §B.1 · pp. 11–12
• Under mixed noise, VaaWIT achieved 59.2 BLEU with 10.2% degradation, while Full FT scored 52.0 BLEU with -15.8% degradation.The reported robustness difference was 5.6 percentage points; under low resolution, degradation was -6.4% for VaaWIT versus -10.7% for Full FT.Source: §B.1 · p. 12

B.2 VAA’s Parameter Effects Analysis

• The controlled ZH-EN configurations shared base architectures; fixed gates retained identical adapter parameters, while VAA dynamically set g=σ(MLP_G(h_g)) from global visual features.The comparisons included no adapter, fixed gates of 1.0 and 0.5, random gates sampled from N(0.5, 0.1), and the complete dynamic-gate approach.Source: §B.2 · p. 12
• +0.3 BLEU (65.0→65.3) and +0.4 COMET came from Fixed Gate (g=1.0) versus w/o VAA; dynamic gating added +0.6 BLEU (65.3→65.9) and +4.6 COMET.The passage attributes 33% of the total gain to adapter parameters alone and 67% to dynamic gating.Source: §B.2 · p. 12
• Random Gate scored 64.5 vs. 65.0 BLEU for w/o VAA, while Fixed Gate (g=0.5) also underperformed the baseline, indicating uninformed or static modulation cannot replace adaptive gating.The passage says arbitrary modulation harms performance and that gating must be visual-aware to provide benefit.Source: §B.2 · p. 12

B.3 Fine-grained Text Fidelity Analysis

• The analysis reports CER and numeric token accuracy on ZH-EN because BLEU and COMET may miss character-level fidelity important for brand names, numeric tokens, and special characters.The evaluation compares VaaWIT with representative baselines.Source: §B.3 · p. 12
• VaaWIT achieves 8.1% CER and 89.7% numeric accuracy, improving over Full Fine-Tuning by 4.2 and 8.5 percentage points, respectively.The comparison is on the ZH-EN task.Source: §B.3 · p. 12
• Zero-shot Qwen3-VL-32B records 28.7% CER and 65.3% numeric accuracy, substantially below VaaWIT on both measures.The passage links VaaWIT’s gains to DSAM’s fine-grained visual fusion, particularly for stylized text, brand logos, and numeric information in e-commerce Web images.Source: §B.3 · p. 12