PaperlayerView exact revision on arXiv

Source-linked AI summary

MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation

Bo Li, Ningyuan Deng, Tianyu Dong, Shaobo Wang, Shaolin Zhu, Lijie Wen

arXiv:2604.16943v1cs.CL

TL;DR

MLLMs struggle with fine-grained textual information in images, while cascaded and standard fine-tuning approaches face limitations in image translation. MNAFT identifies task-relevant language-specific and language-agnostic neurons and selectively fine-tunes them, consistently outperforming existing image-translation methods across datasets and language pairs.

  • Problem

    MLLMs struggle to capture nuanced multilingual text in images, creating a mismatch between visual-text inputs and textual outputs that limits image-translation accuracy and fluency.

  • Method

    MNAFT uses instruction-driven activation analysis and Taylor expansion-based importance scores to identify relevant language-specific and language-agnostic neurons, then selectively fine-tunes their weights and biases while freezing other parameters.

  • Results

    MNAFT consistently outperforms existing state-of-the-art image-translation methods, including cascade pipelines and multiple LLM fine-tuning strategies, across different datasets and language pairs.

  • Takeaways & Limitations

    Selective neuron adaptation preserves general or cross-modal knowledge while concentrating adaptation on neurons relevant to each image-translation task.

  • Takeaways & Limitations

    The current implementation masks gradients only at the optimizer-update stage, so it does not reduce gradient computation itself and may leave further efficiency gains unrealized.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have shown impressive capabilities, yet they often struggle to effectively capture the fine-grained textual information within images crucial for accurate image translation. This often leads to a modality gap between visual text inputs and textual inputs/outputs for image translation. Existing methods, primarily relying on instruction fine-tuning, risk parameter redundancy of pre-trained knowledge, hindering generalization performance. To address this, we introduce modality neuron-aware fine-tuning (MNAFT), a novel approach that takes advantage of the specialized roles of individual neurons within MLLMs for enhanced image translation. MNAFT identifies language-agnostic and language-specific neurons in both vision and language modules through an instruction-driven activation analysis, evaluating their importance in various translation tasks. We then perform selective fine-tuning, updating only the parameters of language-specific and language-agnostic neurons within the selected layers relevant to the target task, while preserving the knowledge encoded in other neurons and layers. Our extensive experiments on multiple benchmarks demonstrate that MNAFT significantly outperforms state-of-the-art image translation methods, including cascaded models, standard full fine-tuning, and parameter-efficient tuning techniques. Furthermore, we provide comprehensive analysis, including visualizations of neuron activations and clustering patterns, to offer insights into the roles of different neuron groups in mediating cross-modal understanding and facilitating accurate language-specific translation.

1 Introduction

Image translation matters for global accessibility, but MLLMs struggle with fine-grained multilingual text because of a modality gap and indiscriminate adaptation. MNAFT addresses these challenges by identifying specialized neurons and selectively fine-tuning them, achieving state-of-the-art results while analyzing their functional roles.

  • Cascaded OCR–MT systems suffer error propagation, sequential-processing inefficiency, and limited holistic context.
  • MLLM visual encoders may inadequately capture nuanced multilingual text, creating a modality gap that reduces translation accuracy and fluency.
  • Uniformly updating MLLM parameters overlooks neuron specialization and risks parameter redundancy during image-translation fine-tuning.
  • MNAFT uses instruction-driven activation and gradient analysis to identify language-specific and modality-shared neurons, then selectively fine-tunes relevant neurons.
  • MNAFT also visualizes neuron specialization during cross-modal and language-specific processing.
  • MNAFT achieves state-of-the-art image-translation performance across comprehensive experiments.

2 Related work

Image translation research has progressed from cascaded OCR–NMT pipelines toward end-to-end multimodal models, while fine-tuning methods address the adaptation cost of MLLMs.

  • Image translation approaches: Cascaded image translation combines OCR extraction with separate neural machine translation, whereas end-to-end systems translate directly from visual input.
  • Image translation approaches: Recent end-to-end methods reduce modality-gap effects, OCR errors, or language-alignment burdens through aligners, multimodal codebooks, and image tokenizers.
  • MLLM adaptation: MLLM-based image translation commonly uses a visual encoder and connector to project visual tokens into the LLM word-embedding space.
  • MLLM adaptation: Full fine-tuning can create parameter redundancy and high computational costs, motivating parameter-efficient methods such as LoRA and DORA.

3 Preliminaries

The paper defines image translation as generating target-language text directly from an image and estimates neuron importance by approximating the loss change caused by removing each neuron.

  • Task definition: Image translation maps an input image containing source-language text directly to its target-language translation.
  • Task definition: The dataset includes an image, its source-language text as auxiliary neuron-analysis information, and the corresponding target translation.
  • Taylor expansion: Taylor expansion estimates the loss effect of setting a neuron activation to zero while holding other neuron activations fixed.
  • MNAFT framework: MNAFT evaluates neuron importance in vision and language layers across languages using instruction-driven analysis.
  • Taylor expansion: The magnitude of the approximated loss change defines neuron importance for selective fine-tuning.

4 Method

MNAFT identifies task- and language-relevant neurons through instruction-driven activation analysis and Taylor expansion-based importance scores, then selectively fine-tunes them within selected vision and language layers.

  • 4.1 Insight: Instruction-driven prompts activate neurons for image-text recognition and language-pair translation, exposing modality- and language-specific specialization.The method uses distinct instructions for recognizing text in images and translating between specified source and target languages.
  • 4.1 Insight: Taylor expansion-based awareness scores estimate each neuron’s importance by measuring loss changes induced by perturbing its output.Scores computed across tasks and language pairs distinguish consistently important language-agnostic neurons from language-specific neurons.
  • 4.2 Neurons identification: Layer relevance is aggregated and ranked to select the most important modules in the vision encoder and language model for optimization.The selected vision and language layers are determined from aggregated neuron relevance scores and corresponding layer-selection hyperparameters.
  • 4.2 Neurons identification: Variance-based thresholds classify neurons as general-language or specific-language units according to their linguistic-awareness scores.Neurons below the threshold are categorized as language-agnostic, whereas those above it are classified as language-specific.
  • 4.3 Selective fine-tuning strategy: MNAFT freezes all parameters except the weights and biases of selected language-agnostic and language-specific neurons in the identified layers.Gradient masks zero updates for unselected neurons, preserving general knowledge while adapting selected parameters to the target task.

5 Experiments

The experiments evaluate MNAFT across six image-translation tasks and four publicly available datasets, comparing it with end-to-end and cascaded baselines and analyzing its components through ablations.

  • 5 Experiments: Six image-translation tasks across four publicly available datasets are used to compare MNAFT with end-to-end and cascaded baselines.The study also conducts ablation experiments to examine the contribution of each MNAFT component.

5.1 Setup

The setup spans synthetic and real image-translation datasets, standardized training and testing subsets, multiple baseline categories, and metrics covering similarity, fluency, and accuracy.

  • Datasets: ECOIT contains 480K paired translation sentences, while MIT-10M contains 840K images covering 14 language pairs.ECOIT is an e-commerce dataset; MIT-10M is a cleaned and multilingual-validated real-world corpus.
  • Data split: Each task uses 100K training items and 100 test items, while neuron identification computes activation- and gradient-based importance scores without parameter updates.The neuron-identification stage is conducted on the test set and functions as structural selection rather than training.
  • Baselines: MNAFT is compared with cascaded OCR-plus-NLLB translation, prompted Qwen2.5-VL-3B, full fine-tuning, parameter-efficient methods, and established end-to-end systems.Parameter-efficient baselines include M2PT, LoRA, DoRA, and MixLoRA; end-to-end comparisons include ItNet, PEIT, Translatotron-V, UMTIT, E2ETIT, and DIMTDA.
  • Evaluation: Evaluation covers semantic similarity, fluency, and accuracy using BLEU and METEOR among the reported metrics.BLEU measures n-gram overlap, whereas METEOR incorporates synonyms and word order.

5.2 Main result

Across six tasks and four datasets, MNAFT consistently outperforms cascaded OCR-plus-MT pipelines, MLLM prompting baselines, and alternative fine-tuning methods. The results indicate robust performance across varied image-translation scenarios.

  • MNAFT consistently achieves superior METEOR and BLEU performance across six tasks and four datasets.Table 1 evaluates cascaded, MLLM baseline, and fine-tuning methods.
  • Comparison to Cascaded Models: MNAFT significantly outperforms EasyOCR NLLB and PP-OCRv3 NLLB on every task.For ECOIT (ZH-EN), MNAFT reaches a METEOR score of 75.1, compared with 13.7 and 13.1 for the cascaded systems.
  • Comparison to MLLM Baselines: MNAFT outperforms Qwen2.5-VL-3B under zero-shot, one-shot, and chain-of-thought prompting.It also exceeds Text-only on most tasks, indicating effective use of visual and textual information.
  • Comparison to Fine-tuning Methods: MNAFT outperforms Full Fine-tuning, LoRA, DoRA, MixLoRA, and M2PT across image-translation datasets.On MIT-10M (DE-EN), its METEOR score rises from 62.6 with Full Fine-tuning to 79.8.

5.3 Comparison with SOTA IT models

Against specialized state-of-the-art image-translation models, MNAFT achieves the best performance on both evaluated datasets. It surpasses all six compared models in METEOR and BLEU, including the strongest competing model on ECOIT (ZH-EN).

  • MNAFT achieves the best METEOR and BLEU results on both ECOIT (ZH-EN) and IIMT (DE-EN).Table 2 compares MNAFT with six state-of-the-art image-translation models.
  • On ECOIT (ZH-EN), MNAFT scores 75.1 METEOR and 54.6 BLEU, exceeding Translatotron-V’s 73.1 and 52.6.The comparison identifies Translatotron-V as the second-best model on this dataset.
  • The results show that MNAFT outperforms existing specialized models across different language pairs.The paper attributes this performance to using MLLM multimodal capabilities while preserving pre-trained knowledge and adapting to image-translation nuances.

5.4 Ablation study

The ablation study shows that MNAFT’s selective neuron choice and joint vision-language adaptation are central to its performance. Specific-neuron tuning is more effective than general-neuron tuning, while the relative value of language and vision layers varies by task.

  • Effect of neuron type: Fine-tuning only general neurons causes a significant performance drop relative to MNAFT.General neurons alone capture broader features but lack the specific knowledge required for accurate translation.
  • Effect of neuron type: Fine-tuning specific neurons performs substantially better than general-neuron tuning and approaches MNAFT, especially on ECOIT.These neurons likely capture target-language or visual-text information needed for translation.
  • Effect of layer selection: Fine-tuning all neurons without selection performs worse than full MNAFT, demonstrating the importance of neuron selection.Selective updating reduces the negative effects of modifying irrelevant parameters.
  • Effect of layer selection: Language-layer tuning is comparable to specific-neuron tuning on OPUS-MIT-5M but performs worse on ECOIT.This suggests that visual-layer adaptation contributes more strongly in some translation tasks.
  • Effect of layer selection: Vision-only tuning produces the lowest ablation performance, while the combined targeted approach yields the best overall results.The study reports that specific-neuron selection and joint vision-language fine-tuning are both important, with layer importance varying by language pair and dataset.

6 Analysis

Activation analyses reveal task-dependent specialization across language and vision layers, while clustering distinguishes specific neurons from broader general-purpose populations. Across models and qualitative cases, MNAFT improves translation quality and reduces training cost, though its current implementation leaves gradient computation unpruned.

  • 6.1 Neuron identification analysis: Task-dependent activation peaks indicate that different language and vision layers participate in distinct image translation tasks.Average activations and layer-to-layer changes vary across tasks, including OPUS-MIT-5M and ECOIT.
  • 6.2 Clustering analysis of general and specific neurons: Language-specific neurons form task and language-related clusters, whereas general language neurons show more mixed representations.DE-EN and FR-EN clusters are closer than ZH-EN and EN-ZH clusters.
  • 6.2 Clustering analysis of general and specific neurons: Vision-specific neurons form less-defined task-related clusters, while general vision neurons diffusely encode broader visual features such as text regions and image layout.The weaker visual clustering suggests less language-specific association than in linguistic features.
  • 6.3 Results on Qwen2.5-VL-7B and LLaVA-NeXT: MNAFT outperforms Full Fine-tuning and LoRA across all reported tasks on both Qwen2.5-VL-7B and LLaVA-NeXT.The same trend across two model architectures supports the reported generalizability and scalability of the method.
  • 6.4 Case study: MNAFT qualitatively improves handling of hallucinated text, advertising nuance, and contextual pronoun references in image translation cases.The cases contrast MNAFT with base-model translations involving product packaging, promotional text, and a headline.
  • 6.5 Computational cost analysis: 24% faster fine-tuning and 22% GPU memory usage versus Full Fine-tuning were reported for ECOIT, while MNAFT was faster than LoRA with moderately higher memory use.ECOIT comparisons were 7.0 h versus 9.2 h and 20.8 GB versus 94.3 GB; LoRA used about 12 GB versus MNAFT’s 21 GB.
  • 6.5 Computational cost analysis: The current implementation masks optimizer updates but does not prune gradient computation, leaving further sparse-backpropagation optimizations for future work.Gradients are still propagated for all activations before updates are restricted to selected neurons.

7 Broader applicability and future work

The paper frames MNAFT’s neuron-aware premise as potentially applicable beyond image translation. It identifies specialized neurons, evaluates their downstream importance, and preserves non-essential parameters while proposing extensions to other multimodal tasks.

  • Broader applicability: MNAFT’s premise is that distinct neuronal populations specialize in sensory inputs, linguistic structures, or inter-modal relationships.The paper presents this functional segregation as relevant to complex neural architectures beyond image-to-text conversion.
  • Methodological extension: The proposed adaptation process identifies neurons sensitive to particular data types or assignments and evaluates their impact on downstream objectives.These steps enable targeted parameter adjustments.
  • Methodological extension: Preserving non-essential neurons and layers is intended to counter catastrophic forgetting and redundant parameter updates during foundation-model adaptation.The paper identifies these as common challenges in adapting large models.
  • Potential applications: Future applications include image captioning, visual question answering, and visually grounded dialogue through task-relevant neuron selection.The paper gives examples involving salient objects, spatial reasoning, query comprehension, dialogue coherence, and sentiment recognition.

8 Conclusion

The conclusion presents MNAFT as a neuron-aware approach that selectively adapts MLLMs for image translation while reducing parameter redundancy. Experiments, ablations, and visual analyses support its effectiveness, and larger-model results motivate future transfer studies across tasks and modalities.

  • 8 Conclusion: MNAFT selectively fine-tunes task-relevant neurons in MLLMs, targeting language-specific and general or cross-modal knowledge while mitigating parameter redundancy.The conclusion reports experiments across datasets and language pairs, including comparisons with cascaded pipelines and other fine-tuning strategies.
  • 8 Conclusion: MNAFT consistently outperforms existing state-of-the-art image translation methods across different datasets and language pairs.Ablations support the neuron-selection mechanism and joint vision-language adaptation.
  • 8 Conclusion: Visualizations provide evidence that different neuron types specialize within the model, supporting the method’s underlying design principles.The conclusion links these analyses to cross-modal and language-specific processing.
  • 8 Conclusion: Results on Qwen2.5-VL-7B and LLaVA-NeXT demonstrate scalability and generalizability across different MLLM architectures.The authors identify transferability to different tasks and modalities as a next research direction.
Loading 2604.16943v1…