Source-linked AI summary
A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, Jiebo Luo
TL;DR
Existing multi-modal NMT models do not fully exploit fine-grained semantic correspondences between textual and visual units. The paper introduces a graph-based multi-modal fusion encoder that represents both modalities in a unified graph and iteratively models their interactions; experiments on Multi30K show superiority over competitive baselines. Future work identifies richer visual attributes, dependency trees, scene graphs, and other multimodal tasks as extensions.
Problem
Existing multi-modal NMT models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, limiting multimodal representation learning.
Method
The encoder builds a unified graph of textual words and visual objects, applies stacked graph-based fusion layers for semantic interactions, and supplies node representations to the decoder.
Results
The model significantly outperforms several competitive baselines, with experiments and analysis on Multi30K indicating effective multimodal information fusion.
Takeaways & Limitations
Fine-grained semantic relationships between multimodal units can be exploited through graph-based encoding for multi-modal NMT.
Takeaways & Limitations
The current work does not yet incorporate visual-object attributes, dependency trees, or scene graphs, and future work plans applications to other multimodal tasks.
Abstract
from arXiv · showhide
Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images. However, dominant multi-modal NMT models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have potential to refine multi-modal representation learning. To deal with this issue, in this paper, we propose a novel graph-based multi-modal fusion encoder for NMT. Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects). We then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, these representations provide an attention-based context vector for the decoder. We evaluate our proposed encoder on the Multi30K datasets. Experimental results and in-depth analysis show the superiority of our multi-modal NMT model.
1 Introduction
The paper addresses the limited use of fine-grained correspondences between textual and visual semantic units in multi-modal NMT by introducing a graph-based fusion encoder. Its unified graph and graph-based fusion layers support semantic interactions, and experiments on Multi30K show stronger performance than competitive baselines.
- Motivation: Fine-grained semantic correspondences between words and visual objects are not fully exploited by existing multi-modal NMT models.The paper identifies unified representation and cross-modal semantic interaction as key challenges.
- Method: The proposed unified multi-modal graph represents words and visual objects as nodes and models intra-modal and inter-modal semantic relationships with edges.This graph is designed to capture relationships within each modality and correspondences across modalities.
- Method: The graph-based multi-modal fusion encoder stacks fusion layers that iteratively perform semantic interactions among graph nodes.The encoder distinguishes the two modalities while conducting graph encoding.
- Evaluation: The model is evaluated extensively on Multi30K datasets covering two language pairs.
- Results: The multi-modal NMT model significantly outperforms several competitive baselines, and analysis indicates effective fusion of multi-modal information.
2 NMT with Graph-based Multi-modal Fusion Encoder
The model encodes sentence–image pairs as a unified graph whose textual and visual nodes exchange intra- and inter-modal information through stacked fusion layers. The resulting multimodal textual states support attention-based decoding in an attentional encoder–decoder framework.
- The model uses an attentional encoder–decoder framework with log-likelihood maximization as its training objective.
- Multi-modal Graph: The input sentence–image pair is represented as an undirected graph whose nodes are textual words or visual objects.Words are included as separate textual nodes, while noun phrases are grounded to detected visual objects.
- Multi-modal Graph: Intra-modal edges connect nodes within the same modality, while inter-modal edges connect noun-phrase text nodes to their corresponding visual nodes.
- Embedding Layer: The embedding layer initializes textual nodes from word embeddings and position encodings, and projects Faster-RCNN visual features into the textual representation space.
- Graph-based Multi-modal Fusion Layers: Stacked graph-based multimodal fusion layers sequentially perform intra-modal self-attention and inter-modal cross-modal gating to update node states.The updates use separate parameters for textual and visual nodes and combine contextual and cross-modal information.
- Decoder: The decoder attends only to textual node states because the fusion layers have already incorporated visual information into them.It follows a Transformer-style stack with masked self-attention, encoder–decoder attention, and position-wise feed-forward networks.
3 Experiment
Experiments on Multi30K evaluate the graph-based encoder across translation tasks, layer settings, model comparisons, translation groups, ablations, and efficiency. The model generally outperforms baselines, with three fusion layers selected and gains supported by explicit cross-modal interactions.
- Setup: Experiments use Multi30K for English⇒German and English⇒French translation, with additional WMT17 and ambiguous MSCOCO test sets.Training, validation, and test splits contain 29,000, 1,014, and 1,000 instances; WMT17 and MSCOCO contain 1,000 and 461 instances.
- Layer number: The model achieves its best validation performance with Le=3 graph-based multi-modal fusion layers.The selected setting is used in all subsequent experiments.
- Main results: On En⇒De, the proposed model outperforms most existing models and all baselines, while remaining comparable to Fusion-conv(RNN) and Trg-mul(RNN) on METEOR.The comparison is reported in Table 1.
- Grouped results: The model consistently achieves the best BLEU scores across groups divided by source sentence length and noun-phrase count.Improvements over baselines are more significant for sentences with more phrases, which are usually longer.
- Efficiency: The model processes approximately 1.1K tokens per second during training and 16.7 sentences per second during decoding, with only a small number of extra parameters.Training speed is comparable to other multi-modal baselines, while decoding is slightly slower than Transformer.
- Ablation study: Ablations show significant degradation when inter-modal fusion is removed or visual grounding is replaced with fully connected correspondences.The fully connected design introduces noise, while removing inter-modal fusion eliminates semantic interactions between modalities.
4 Related Work
Prior multimodal work incorporated visual features through attention and multimodal fusion, while this paper represents sentence-image pairs as unified graphs to capture semantic relationships. Its approach extends graph neural networks to multimodal NMT rather than building separate graphs for each modality.
- Earlier multimodal NMT models incorporated global or regional visual features through attention-based fusion mechanisms.
- Visual information may be ignored by multimodal NMT models, although limited textual context can improve its use.
- The proposed approach represents each sentence-image pair as a unified graph that captures semantic relationships between multimodal units.
- Unlike ObjectAsToken(TF), the model first learns within-modality context and assigns different encoding parameters to different modalities.
- Unlike LXMERT, the model performs self-attention and cross-modal gating sequentially at each layer for multimodal NMT.
- For GNN-based multimodal learning, this work uses one unified multimodal graph rather than an individual graph for each modality.
5 Conclusion
The paper proposes a graph-based multi-modal fusion encoder for NMT that exploits semantic relationships between multimodal units. Experiments on Multi30K demonstrate the model’s effectiveness, while future work targets richer graphs and additional multimodal tasks.
- The proposed graph-based multi-modal fusion encoder exploits semantic relationships between multimodal semantic units for NMT.
- Experiments and analysis on the Multi30K dataset demonstrate the effectiveness of the model.
- Future work will enrich multi-modal graphs with visual-object attributes, dependency trees, and scene graphs.
- The authors also plan to apply the model to other multimodal tasks, including multimodal sentiment analysis.