PaperlayerView exact revision on arXiv

Source-linked AI summary

VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation

Bo Li, Ronghao Chen, Ningyuan Deng, Huacan Wang, Shaolin Zhu, Lijie Wen

arXiv:2605.24675v1cs.CVcs.AI

TL;DR

Web image translation requires fine-grained character recognition alongside multilingual reasoning, yet standard visual encoders emphasize high-level semantics. VaaWIT addresses this with dual-stream attention and a visual-aware adapter, and experiments across eight tasks and three benchmarks show stronger performance than open-source SOTA baselines with competitiveness against proprietary models. The paper concludes that structured visual fusion with lightweight LLM adaptation supports complex cross-modal translation while maintaining parameter and training efficiency.

  • Problem

    Web image translation requires simultaneous OCR, translation, and visual-context preservation, while standard encoders often miss the fine-grained character details needed for diverse text morphologies.

  • Method

    VaaWIT uses bidirectional Dual-Stream Attention to fuse multilingual semantic and detailed visual features, then dynamically injects them into a frozen LLM through a Visual-Aware Adapter.

  • Results

    Experiments on eight translation tasks in three benchmarks show that VaaWIT significantly outperforms open-source SOTA baselines and achieves performance competitive with proprietary commercial models.

  • Takeaways & Limitations

    The results support combining structured visual feature fusion with lightweight LLM adaptation for complex cross-modal tasks while maintaining parameter and training efficiency.

  • Takeaways & Limitations

    The end-to-end formulation assumes the model can implicitly perform OCR and translation simultaneously from pixel-level information.

Abstract

from arXiv · show

Translating text embedded in Web images is crucial for improving content accessibility and cross-lingual information retrieval, particularly within social media and e-commerce domains. Although Large Vision-Language Models (LVLMs) have advanced multimodal understanding, applying them to Web image translation remains challenging due to the visual representation gap: standard encoders often prioritize high-level semantics over the fine-grained visual details required for recognizing diverse character morphologies. To address this challenge, we propose VaaWIT, an end-to-end framework that adapts Large Language Models for multilingual Web image translation. The framework introduces two key technical contributions: (1) a Dual-Stream Attention Module (DSAM), which facilitates bidirectional interaction between multilingual semantic features and detailed visual representations, thereby synthesizing unified features robust to textual variations; and (2) a Visual-Aware Adapter (VAA), a parameter-efficient fine-tuning strategy that dynamically injects these fused visual cues into the frozen LLM backbone. This design enables the model to align the visual context with linguistic reasoning effectively while minimizing computational costs. Extensive experiments on eight tasks on three public benchmarks demonstrate that VaaWIT significantly outperforms state-of-the-art (SOTA) open-source baselines and achieves competitive performance against proprietary models. These results validate the efficacy of integrating fine-grained visual perception into LLMs for complex Web content analysis.

1 Introduction

Web image translation must recognize and translate embedded text while preserving visual-scene context, but standard encoders often miss fine-grained character details. VaaWIT addresses this gap with dual-stream visual fusion and visual-aware LLM adaptation.

  • Motivation: Web image translation jointly requires OCR, translation, and preservation of visual-scene semantics across diverse fonts, layouts, and backgrounds.The task supports content accessibility and cross-lingual information retrieval.
  • Existing limitations: Cascaded OCR-plus-NMT systems propagate recognition errors, while specialized end-to-end models often lack sufficient scale and generalized Web knowledge.These limitations motivate adapting large language models for the task.
  • Visual representation gap: Standard visual encoders prioritize image-level semantics over the fine-grained character morphology needed to recognize text such as “Sale 50%” in images.This visual representation gap limits off-the-shelf LVLMs for Web image translation.
  • VaaWIT: VaaWIT combines semantic and detail visual streams through bidirectional cross-attention, then injects fused cues into a frozen LLM using a lightweight Visual-Aware Adapter.The framework is organized around dual-stream encoding, visual feature fusion, and visual-aware LLM adaptation.
  • Results: Extensive experiments on eight translation tasks across three benchmarks show that VaaWIT significantly outperforms open-source SOTA baselines and remains competitive with proprietary models.The reported results support the effectiveness of its visual-aware adaptation strategy.

2 Preliminaries

VaaWIT formulates Web image translation as autoregressive multimodal generation from image pixels to target tokens, requiring implicit OCR and translation. Its visual pipeline uses complementary semantic and detail representations to support this objective.

  • Problem formulation: The training dataset pairs raw Web images containing embedded text with target translation sequences of length L.The model learns a multimodal mapping conditioned on visual input.
  • Problem formulation: VaaWIT maximizes the autoregressive log-likelihood of target tokens generated from the visual input.Unlike standard machine translation, the end-to-end setting starts from pixel-level information.
  • Problem formulation: The end-to-end formulation requires the model to implicitly perform OCR and translation simultaneously using only pixel-level information.This distinguishes the task from conventional source-text machine translation.
  • Visual encoding: A multilingual semantic encoder captures high-level textual concepts, while a separate detail encoder is introduced to compensate for lost high-frequency spatial information.The two backbones provide complementary representations for semantic understanding and character recognition.

3 Methodology

VaaWIT bridges fine-grained visual perception and multilingual reasoning through dual-stream feature fusion and visual-aware adaptation of a frozen LLM. Its training progressively aligns visual features and jointly optimizes semantic matching, text translation, and image translation.

  • Framework Overview: VaaWIT uses an end-to-end pipeline combining dual-stream visual encoding, visual feature fusion, and visual-aware LLM adaptation.The design targets unified visual-linguistic processing for Web image translation.
  • Dual-Stream Attention Module: DSAM projects semantic and detail features into a shared space, then uses bidirectional attention to refine each stream with information from the other.SGDR retrieves morphological details using semantic queries, while DISR enriches semantic features with precise visual cues.
  • Dual-Stream Attention Module: The refined streams are concatenated and passed through an MLP to produce LLM-dimensional visual representations containing linguistic context and character-level visual details.The fused representation is intended to support both translation reasoning and character recognition.
  • Visual-Aware LLM Adaptation: VAA aggregates dense fused visual patches into a global context vector and uses a content-dependent gate to regulate visual information within transformer layers.A bottleneck adapter and residual connection selectively modify the frozen LLM while preserving pretrained linguistic knowledge.
  • Visual-Aware LLM Adaptation: The gated adapter preserves pretrained linguistic knowledge while enabling visual-aware reasoning with parameter-efficient updates.Only the lightweight adapter weights and gating network are updated during training.
  • Training: Training freezes the visual encoders and LLM backbone while updating DSAM and VAA through visual-language alignment followed by multi-task joint learning.The joint objective combines image-text matching, text translation learning, and image translation learning, with weights prioritizing end-to-end translation performance.

4 Experiments

VaaWIT is evaluated across eight multilingual Web image translation tasks and three public datasets against cascaded systems, LVLMs, adaptation strategies, and specialized models. It achieves strong translation quality while using substantially fewer trainable parameters than full fine-tuning, with ablations supporting the contributions of DSAM and VAA.

  • Experimental Setup: Evaluation covers 8 tasks across 3 public Web image translation datasets, using BLEU and COMET against cascaded systems, LVLMs, and tuning strategies.The tasks span MIT-10M, ECOIT, and OPUS-MIT-5M, with Qwen3-8B and LLaMA3.1-8B backbones also tested.
  • Main Results: VaaWIT improves over traditional cascade models across all language pairs and exceeds EasyOCR-plus-translation baselines by more than 50 BLEU points on ZH-EN.The comparison attributes this result to the end-to-end design, which avoids OCR-to-translation error propagation and uses multimodal context.
  • Main Results: VaaWIT achieves performance comparable to GPT4.1 and Gemini2.5 Pro across most tasks and surpasses them on several tasks.The results also report substantially stronger performance than the evaluated zero-shot SOTA LVLMs.
  • Efficiency: With approximately 50M trainable parameters, VaaWIT surpasses fully fine-tuned models on most tasks and improves over Full FT by 7.6 BLEU and 2.9 COMET on IT-EN.VaaWIT updates only DSAM and VAA while keeping the visual encoders and LLM backbone frozen.
  • Comparison with Image Translation Models: VaaWIT reaches 65.9 BLEU and 94.8 COMET on ZH-EN, surpassing Translatotron-V by 13.3 BLEU and 11.7 COMET.On EN-IT, it achieves 38.1 BLEU and 85.9 COMET, leading the compared image translation models.
  • Ablation Study: Removing DSAM and VAA reduces performance, while removing both causes a ZH-EN decrease of 4.6 BLEU and 6.8 COMET.The ablation results associate DSAM with visual-representation informativeness and VAA with integrating visual information into the LLM.

5 Analysis

The analyses show that VaaWIT’s explicit visual-feature fusion and uniform visual-aware adaptation are central to its performance, while the framework remains effective across encoders, training settings, and complex image cases.

  • VAA Gating Strategies: Global gating offers a resource-efficient balance among performance, parameter efficiency, and inference speed in the evaluated IT-EN comparison.The comparison evaluates global, layer-specific, token-dependent, and combined layer-plus-token gating strategies.
  • Vision Feature Fusion: DSAM improves nearly 3 BLEU over Simple Concat on both ZH-EN and HI-EN, while Interleaving Fusion and Self-Attention remain below DSAM.The result indicates that basic feature combination does not fully exploit complementary visual information.
  • Bidirectional Interaction: DSAM exceeds parameter-matched one-way cross-attention by 1.4–1.7 BLEU and 3.8–4.1 COMET on ZH-EN, with both one-way variants also trailing on HI-EN.The comparison supports explicit bidirectional interaction between semantic and detailed visual features.
  • VAA Insertion Strategies: Early-only VAA insertion lowers BLEU by 2.1 points on JA-EN and 2.3 points on HI-EN versus uniform insertion, while late-only insertion also degrades performance.Adapters are inserted after the FFN sub-layer in all layers for the uniform strategy; early and late variants have matched parameter counts.
  • Visual Encoder Combinations: Dual visual encoders outperform single-encoder baselines, and mSigLIP plus DINOv2 achieves the best overall performance among the evaluated combinations.The analysis identifies complementary visual information as important for recognizing stylized or noisy text in Web scenes.
  • Training Analysis: Minor loss-weight changes produce only small BLEU and COMET differences, whereas skipping Stage 1 reduces performance by 2.8 BLEU and 3.4 COMET.The full two-stage pipeline reaches 65.9 BLEU / 94.8 COMET, compared with 63.1 / 91.4 when Stage 1 is skipped.
  • Case Study: In an EN-IT e-commerce case, VaaWIT translates all textual information and uses visual context to translate “UMIDO” as “Organic,” unlike GPT4.1’s omissions.The case concerns scattered text combining a brand logo and descriptive content.

6 Related work

Related work includes cascaded OCR–MT pipelines and unified end-to-end image translation models. VaaWIT builds on this line by combining bidirectional visual fusion with parameter-efficient, context-aware LLM adaptation.

  • Cascaded Systems: Cascaded Web image translation systems combine OCR with machine translation, but OCR errors propagate into erroneous translations.End-to-end models were developed partly to address this fragility.
  • End-to-End Models: Existing end-to-end models explore multi-task learning, knowledge distillation, modality alignment, and multimodal representation learning, but remain limited by scale and diversity.These models provide more unified approaches than cascaded pipelines.
  • VaaWIT: VaaWIT addresses the task through DSAM’s bidirectional cross-attention and VAA’s parameter-efficient, context-aware dynamic gating of LLM adaptation.Together, the modules target the visual representation gap in multilingual Web image translation.

7 Conclusion

VaaWIT is a framework for multilingual Web image translation that combines structured visual-feature fusion with dynamic, lightweight LLM adaptation. Experiments report performance above previous methods and comparable to commercial systems while maintaining parameter and training efficiency.

  • Conclusion: VaaWIT combines DSAM for deep interaction between complementary visual features with VAA for parameter-efficient dynamic adaptation inside the LLM backbone.The framework is designed specifically for multilingual Web image translation.
  • Conclusion: Experiments report that VaaWIT outperforms previous methods and achieves performance comparable to commercial systems.The conclusion also emphasizes parameter and training efficiency in complex cross-modal tasks.
  • Conclusion: Future work could extend VaaWIT with layout-aware translation.

A Implementation Details

The implementation uses frozen mSigLIP and DINOv2 visual encoders and evaluates VaaWIT on three public Web image translation datasets.

  • Implementation: Experiments use CentOS 7.5, Python 3.9.12, NVIDIA H20 GPUs, CUDA 12.2, and specified PyTorch, torchvision, and Transformers versions.The visual encoders are mSigLIP and DINOv2, both kept frozen during training.

A.2 Two-Stage Training Settings

VaaWIT uses a two-stage training process to align fused visual representations with textual semantics and then evaluate the adapted model under standardized protocols. Training updates only DSAM and VAA while keeping the visual encoders and LLM backbone frozen.

  • Training stages: Stage 1 trains DSAM and VAA on image content and source-language text to align fused visual representations with the LLM embedding space.The stage uses complete training data from MIT-10M, ECOIT, and OPUS-MIT-5M with autoregressive language modeling loss.
  • Training configuration: Only DSAM and VAA, totaling approximately 50M parameters, are updated, while the visual encoders and LLM backbone remain frozen.DeepSpeed ZeRO Stage 2 and gradient accumulation support data-parallel training on 8 NVIDIA H20 GPUs.
  • Evaluation protocol: All methods use the same test sets, metrics, and decoding settings to ensure fair and reproducible comparisons.Zero-shot LVLMs use a unified translation prompt, original-resolution images, greedy decoding, and disabled OCR or tool use.

B.1 Robustness Analysis

The robustness analysis evaluates VaaWIT on degraded Web images using five simulated noise conditions. VaaWIT outperforms the baseline across individual degradations and retains stronger performance under mixed noise.

  • Evaluation scope: The evaluation targets Web-image degradations caused by compression, low resolution, motion blur, and partial occlusion.It uses the ZH-EN task on the MIT-10M test set to assess practical robustness.
  • Noise conditions: Five conditions simulate Gaussian blur, JPEG compression, low resolution, occlusion, and mixed noise.Mixed noise applies all listed degradations simultaneously to represent a challenging Web scenario.
  • Results: VaaWIT shows superior robustness to the baseline under every individual noise type.The reported comparison covers all five simulated degradation settings.
  • Results: Under low resolution, VaaWIT degrades by 6.4%, compared with 10.7% for Full FT.Under mixed noise, VaaWIT maintains 59.2 BLEU with 10.2% degradation.

B.2 VAA’s Parameter Effects Analysis

The parameter-effects analysis separates the contribution of adapter parameters from VAA’s dynamic gating mechanism. Results indicate that visual-aware adaptive modulation contributes more to performance than parameter addition alone.

  • Experimental design: The experiments compare four configurations with identical base architectures but different gating strategies on the ZH-EN task.The design isolates dynamic gating from the effect of adding trainable parameters.
  • Dynamic gate: Dynamic gating adapts the modulation factor from global visual features in the complete VAA approach.The gate is defined as g = σ(MLP_G(h_g)).
  • Parameter effects: +0.3 BLEU (65.0→65.3) and +0.4 COMET from Fixed Gate (g = 1.0) over w/o VAA show the limited benefit of adapter parameters alone.These gains account for 33% of the total performance gain.
  • Dynamic-gating effects: +0.6 BLEU (65.3→65.9) and +4.6 COMET from VaaWIT over Fixed Gate (g = 1.0) represent 67% of the total gain.The comparison identifies dynamic gating as the primary contributor to VAA’s effectiveness.
  • Gating strategies: Random or fixed modulation underperforms visual-aware adaptive modulation, with Random Gate reaching 64.5 versus 65.0 BLEU for w/o VAA.The results indicate that arbitrary or static modulation cannot substitute for context-dependent gating.

B.3 Fine-grained Text Fidelity Analysis

The fine-grained fidelity analysis evaluates character recognition and numeric-token translation, which BLEU and COMET may not fully capture. VaaWIT achieves stronger fidelity than Full Fine-Tuning and zero-shot Qwen3-VL-32B.

  • Evaluation metrics: The analysis reports Character Error Rate and numeric token accuracy for ZH-EN image translation.These metrics target fidelity for brand names, numeric tokens, and special characters beyond broad translation quality.
  • Baseline comparison: VaaWIT records 28.7% CER and 65.3% numeric accuracy for zero-shot Qwen3-VL-32B as the comparison baseline.The reported results associate fine-grained visual fusion with improved recognition of stylized text, brand logos, and numeric information.
Loading 2605.24675v1…