Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting

• Image context can help resolve ambiguities in translation, yet MMT models are not guaranteed to rely on it.Source: §1 · p. 1
• CoMMuTE evaluates disambiguation by asking models to select the correct translation among alternatives, addressing a limitation of traditional metrics such as BLEU.Source: §1 · p. 1
• The method identifies target tokens that benefit from an image and increases their training loss, using PCXMI and its incongruent-image modification.Source: §1 · p. 2
• Metric-based Loss Weighting improves CoMMuTE accuracy on most tested scenarios, by more than 7 percentage points over standard fine-tuning at best.Source: §1 · p. 2
• The authors report that combining PCXMI and Congruency-based PCXMI yields the best results, and that BLEU does not capture models’ visual reliance.Source: §1 · p. 2

• MMT research has used specialized architectures for feature extraction, cross-modal fusion and alignment, and cross-modal attention.Source: §2 · p. 2
• Other approaches include auxiliary objectives, source masking, and data augmentation through generated images or backtranslation.Source: §2 · pp. 2–3
• Multi30k systems’ BLEU gains have been attributed to regularization, and replacing relevant images with random ones did not produce the expected performance drop.Source: §2 · p. 3
• Research has shifted from models trained from scratch toward pretrained models and MLLMs, which can incorporate visual information without architectural modifications.Source: §2 · p. 3
• Metric-based fine-tuning and loss weighting were previously used for text-only context-aware MT; this work combines and adapts them to image-guided MT.Source: §2 · p. 3
• The work also introduces a metric that compares probabilities with correct and incongruent images rather than comparing image context with no image.Source: §2 · p. 3

3 Method

• The method first measures how token probabilities change across image contexts, then increases the loss for context-dependent target tokens during training.It applies token-level weighting to the standard Negative Log Likelihood loss.Source: §3 · pp. 3–4
• PCXMI compares each target token’s probability with and without the accompanying image, requiring the model to translate in both settings.The image is treated as context in this adaptation of PCXMI.Source: §3 · p. 3
• CPCMXI compares token probabilities under correct and incongruent images, and can be used when a model always requires an image.The incongruent image is randomly sampled from the training dataset.Source: §3 · pp. 3–4
• The final token dependence metric takes the maximum of PCXMI and CPCMXI to improve reliability when either metric misses a context-dependent token.Both metrics are calculated for each training example.Source: §3 · p. 4
• The weight function uses the token’s dependence metric, with hyperparameters α and τ controlling the loss increase and threshold, respectively.The weighted loss uses each training example’s token-level metric and the model’s output probability.Source: §3 · p. 4
• The method relies on the model to identify image-dependent tokens, biasing emphasis toward tokens for which it already exhibits strong visual sensitivity.The loss penalizes mistakes more for tokens that require context.Source: §3 · p. 4

4 Experiments

• The experiments fine-tuned Qwen3-VL 4B, Qwen3-VL 8B, and IDEFICS 9B with LoRA on Multi30k for English-to-German, English-to-French, and English-to-Czech.Evaluation used BLEU and COMET on Multi30k test sets and CoMMuTE accuracy, averaged over three runs with standard errors reported.Source: §4 · pp. 4–5, §4.1 · p. 6
• Compared methods included standard fine-tuning, VMLM, Word Dropout, Source Degradation, Metric-based Fine-tuning, and Metric-based Loss Weighting.Metric-based Fine-tuning selected top-k examples using the maximum PCXMI or CPCXMI value after initial fine-tuning; using both metrics performed better in preliminary experiments than either alone.Source: §4 · p. 4

4.1 Results

• For English-to-German, only Word Dropout and Metric-based Loss Weighting recovered and surpassed baseline visual grounding; each technique improved baseline performance in the other two directions.The authors hypothesize that German-English linguistic proximity may reduce reliance on visual context during fine-tuning, but leave this for future work.Source: §4.1 · p. 6
• VMLM showed no large visual-sensitivity gains over other methods in this setup, which used only parallel multimodal data; the authors expect larger gains with large-scale monolingual multimodal data.VMLM used twice as much computation as the other methods, and scaling it further was outside this work's scope.Source: §4.1 · pp. 6–7
• Metric-based Loss Weighting ranked first in CoMMuTE accuracy in 6 of 9 settings and second in the other three; the top methods were not significantly different from each other.The highest results were significantly different from baseline models and Fine-tuning, but low run counts prevented significant differences among top methods.Source: §4.1 · p. 7
• Higher CoMMuTE accuracy did not correlate well with BLEU and COMET, so general translation quality and visual grounding were not necessarily aligned.The authors note that Multi30k test examples may not require images for good translation.Source: §4.1 · p. 7

5.1 Metrics Analysis

• The two metrics sometimes disagree on which target tokens depend on images, so taking their maximum can help when either metric misses a token.Figure 2 presents per-token PCXMI and CPCXMI values for two English-to-German CoMMuTE examples, marking ambiguous source and target words in red.Source: §5.1 · p. 7, Figure 2 · p. 8
• PCXMI and CPCXMI can identify image-dependent disambiguation tokens, but may miss or misidentify them; taking their maximum partially mitigates this.Metric values are concentrated near zero with a long high-value tail, and vary more by model than by language direction.Source: §5.1 · p. 7

5.2 Qualitative Analysis

• The largest example-level BLEU differences mostly involve acceptable translations that differ from the gold target, rather than clearly unacceptable outputs.The comparison focuses on Qwen3-VL 8B English-to-French on MSCOCO, where the method’s BLEU difference from fine-tuning was largest.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3 Combining Weighting with Dropout

• The combined method raises CoMMuTE accuracy for English-to-German and English-to-Czech, but not English-to-French, while reducing BLEU for all tested languages.The BLEU reduction was not observed in COMET; the results suggest the methods can work in tandem, with outputs moving away from gold translations.Source: §5.3 · p. 8, Table 4 · p. 8
• Word Dropout is applied to the source sentence only when no token exceeds threshold τ, combining source-side dropout with target-side loss weighting.The combined method improved CoMMuTE accuracy on English-to-German and English-to-Czech, but did not improve weighting on English-to-French.Source: §5.3 · pp. 7–8

• Metric-based loss weighting increases loss on tokens that benefit from images, identified using PCXMI and CPCXMI based on model outputs with and without visual context.Source: §6 · p. 8
• The method was tested against existing fine-tuning approaches on three MLLMs and three language directions, with increased CoMMuTE accuracy.Source: §6 · p. 8
• CoMMuTE accuracy increased while strong overall translation quality was maintained, and BLEU and COMET did not capture how strongly models rely on visual input.Future work will explore using the metrics for source-side masking and word dropout.Source: §6 · p. 9

• The tested language directions cover only Indo-European sub-families using the Latin script, reflecting the language coverage of Multi30k and CoMMuTE.Source: §7 · p. 9
• Metric-based Loss Weighting uses image-dependence metrics calculated from baseline models; updating them during training or using a more visually sensitive model was not tested.Source: §7 · p. 9
• The tested MLLMs were all under 10B parameters, and the study was limited to PCXMI and CPCXMI despite the possibility that broader metric exploration could improve performance.Source: §7 · p. 9

A Training Details

• VMLM training takes roughly twice as long per epoch as fine-tuning, while other methods have comparable training times.Additional preprocessing and post-processing do not meaningfully increase training time; Metric-based Fine-tuning time covers only its second stage.Source: A Training Details · p. 12, Table 5 · p. 13
• Training used Multi30k, with evaluation on its testsets and the CoMMuTE contrastive dataset.The dataset sizes are reported in Table 6.Source: A Training Details · p. 13, Table 6 · p. 13
• Generation uses temperature 0.7 sampling for Qwen3-VL models and greedy decoding for IDEFICS 9B.The translation and VMLM prompts are provided in Listings 1 and 2.Source: A Training Details · p. 13
• Metric-based Loss Weighting tunes threshold τ by model family and uses constant loss weighting α = 8.The tested thresholds are {5, 6, 7} for Qwen3-VL and {1, 1.5, 2} for IDEFICS 9B.Source: A Training Details · p. 14
• Hyperparameters were selected by first retaining runs within 0.5 BLEU of the best Multi30k validation score, then choosing the run with highest CoMMuTE validation accuracy.Because CoMMuTE validation is available only for English-to-French, the French target side was translated into German and Czech for validation.Source: A Training Details · p. 14

B Ablation Study

• Using the maximum of PCXMI and CPCXMI yields higher CoMMuTE accuracy than using either metric separately.The ablation tests the two metrics individually and their maximum.Source: B Ablation Study · p. 14
• Increasing α raises CoMMuTE accuracy but can reduce BLEU and COMET when set too high.The α values tested are {2, 4, 8, 16}.Source: B Ablation Study · p. 14
• Higher τ values yield higher BLEU and COMET but lower CoMMuTE accuracy, as fewer tokens receive increased loss.Thresholds tested are {4, 5, 6, 7}.Source: B Ablation Study · p. 14

C Extended Metrics Analysis

• The metrics identify image-dependent tokens in selected CoMMuTE target sentences.Figure 7 shows per-token PCXMI and CPCXMI values for Qwen3-VL 4B English-to-German examples.Source: C Extended Metrics Analysis · p. 14
• For Qwen3-VL models, CPCXMI values are more concentrated near zero than PCXMI; IDEFICS 9B shows similar distributions across metrics.Figure 5 summarizes maximum token-level metric values per Multi30k training example.Source: C Extended Metrics Analysis · p. 14
• Figure 4 evaluates BLEU, COMET, and CoMMuTE accuracy across metric choice, α, and τ for Qwen3-VL 4B English-to-German.Results are averaged over three runs, with standard-error bars.Source: Figure 4 · p. 15
• Maximum PCXMI/CPCXMI values are less concentrated near zero on CoMMuTE than on Multi30k, consistent with ambiguous words in every CoMMuTE example.Figure 6 compares the two datasets.Source: C Extended Metrics Analysis · p. 15

D Extended Qualitative Analysis

• Among the 20 largest BLEU drops for weighting, 30% reflect different forms, 45% equivalent translations, and 25% errors; for fine-tuning's largest drops, the split is 10%, 50%, and 40%.In these selected Qwen3-VL 8B English-to-French MSCOCO examples, the largest BLEU drops mostly correspond to acceptable translations differing from the gold target.Source: D Extended Qualitative Analysis · pp. 15–16

E Extended Results

• The combined Metric-based Loss Weighting and Word Dropout results cover all testsets for Qwen3-VL 4B in English-to-German and English-to-French.Table 9 reports averages over three runs.Source: E Extended Results · p. 16, Table 9 · p. 17
• One illustrated English-to-German example translates “bugs” as “Insekten.”Figure 7 presents selected CoMMuTE examples with per-token metric values; Figures 8 and 9 show selected MSCOCO cases where each method scores higher in BLEU.Source: E Extended Results · p. 18, Figure 7 · p. 18, Figure 8 · p. 19