Source-linked AI summary
Multilingual Universal Sentence Encoder for Semantic Retrieval
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil
TL;DR
The paper addresses multilingual sentence retrieval by introducing Transformer- and CNN-based encoders that map 16 languages into one semantic space through multi-task dual-encoder training. Across semantic retrieval, bitext retrieval, question answering, and transfer tasks, the models are competitive with strong systems, and cross-lingual performance approaches monolingual performance for many language pairs.
Problem
Multilingual retrieval requires sentence embeddings that capture semantic similarity across languages in a shared representation space.
Method
The paper trains Transformer and CNN multilingual sentence encoders with a multi-task dual-encoder framework using translation-based and other bridge tasks.
Results
The models achieve good performance on SR, BR, and ReQA, approach monolingual transfer performance, and approach monolingual cross-lingual SR and ReQA performance for many language pairs.
Takeaways & Limitations
The multilingual models provide publicly available sentence and QA retrieval interfaces for embedding text from 16 languages into a shared semantic space.
Takeaways & Limitations
Bitext retrieval performance is degraded for some languages because one SentencePiece vocabulary must cover all 16 languages, especially character-rich Chinese, Korean, and Japanese.
Abstract
from arXiv · showhide
We introduce two pre-trained retrieval focused multilingual sentence encoding models, respectively based on the Transformer and CNN model architectures. The models embed text from 16 languages into a single semantic space using a multi-task trained dual-encoder that learns tied representations using translation based bridge tasks (Chidambaram al., 2018). The models provide performance that is competitive with the state-of-the-art on: semantic retrieval (SR), translation pair bitext retrieval (BR) and retrieval question answering (ReQA). On English transfer learning tasks, our sentence-level embeddings approach, and in some cases exceed, the performance of monolingual, English only, sentence embedding models. Our models are made available for download on TensorFlow Hub.
1 Introduction
The paper introduces multilingual USE sentence-embedding models designed to capture semantic similarity across 16 languages, alongside a retrieval-question-answering interface.
- Three new USE family members comprise multilingual CNN and Transformer models plus a retrieval question-answering interface.The two multilingual models target multilingual semantic similarity, while the third provides an alternative interface to the multilingual Transformer.
- The multilingual models support 16 languages, with Simplified and Traditional Chinese treated as separate languages.The language inventory is presented in Table 1, and the model distinguishes zh from zh-tw because of character-set differences.
2 Model Toolkit
The toolkit provides multilingual sentence-encoding and question-answering interfaces implemented in TensorFlow Hub. Its QA interface incorporates context into response embeddings for efficient nearest-neighbor retrieval.
- The multilingual models are implemented in TensorFlow and publicly distributed as TensorFlow Hub modules.The toolkit includes downloadable saved graphs and example notebooks or documentation through TensorFlow Hub.
- The question-answering interface encodes responses with additional context so answers have high dot-product similarity to their questions.Context may come from surrounding text or a longer answer, supporting answer retrieval from indexed candidates.
- The resulting embeddings support retrieval of indexed candidates with efficient nearest-neighbor search.The paper names FAISS, Annoy, and FLANN as examples of efficient search tools.
3 Encoder Architecture
The encoders use a shared multi-task dual-encoder framework with specialized task layers and offer Transformer and CNN architectures that trade accuracy against resource efficiency.
- A shared encoder is trained on question-answer prediction, translation ranking, and natural language inference tasks.Question-answering and NLI receive additional task-specific hidden layers after the shared encoder.
- The models use a single 128k SentencePiece vocabulary trained on balanced data from all 16 supported languages.Character coverage exceeds 99% for every language, leaving less than 1% of output tokens out of vocabulary.
- The Transformer targets higher accuracy at greater resource cost, whereas the CNN is designed for efficient inference with reduced accuracy.Both architectures average token-level representations into fixed-length sentence embeddings, with the CNN applying additional feedforward layers afterward.
- Transformer: The Transformer computes context-aware token representations with bidirectional self-attention before averaging them into a sentence embedding.The representation incorporates token ordering and identity.
- CNN: The CNN processes token embeddings with convolutional layers and average pooling to produce a fixed-length sentence representation.Additional feedforward layers transform the pooled representation into the final sentence embedding.
4 Training and Configuration
Training combines mined multilingual question-answer and translation pairs with SNLI, using translation to balance language coverage. The released models are configured for separate CNN and Transformer sequence lengths and evaluated against an English semantic-retrieval reference.
- Training data: Training data combines mined question-answer pairs, mined translation pairs, and the English SNLI corpus.Question-answer pairs come from online forums and QA websites, while translation pairs are mined with a system similar to prior work.
- Training data: SNLI is translated into the other 15 languages, and some question-answer pairs are translated so every language has at least 60M training pairs.The multilingual data are balanced across languages despite uneven availability of mined examples.
- Model configuration: CNN inputs are truncated to 256 tokens, while Transformer inputs are truncated to 100 tokens.The CNN uses two convolutional layers with filter widths [1, 2, 3, 5] and filter size 256.
- Evaluation: English semantic-retrieval results are reported as MAP@100 and compared with Gillick et al. models without in-domain training data.The comparison is presented in Table 2.
- Model configuration: The Transformer configuration uses six layers, eight attention heads, hidden size 512, and filter size 2048.Hyperparameters are tuned on development data sampled from the same sources as training data.
5 Experiments on Retrieval Tasks
The models are evaluated on semantic retrieval, bitext retrieval, and retrieval question answering, including cross-lingual settings. Results show strong retrieval performance, with USETrans generally outperforming USECNN and cross-lingual performance remaining close to English-only performance despite greater difficulty.
- Semantic Retrieval (SR): Semantic retrieval identifies all corpus sentences semantically similar to each query, using transitive-closure datasets from Quora and AskUbuntu and MAP evaluation.Both datasets are English only.
- Bitext Retrieval (BR): Bitext retrieval uses 86,000 English–translation document pairs across five languages, yielding 11.3 million aligned sentence pairs per language pair.The evaluated languages are French, Spanish, Russian, Arabic, and Chinese.
- Bitext Retrieval (BR): USETrans is generally better than USECNN on UN bitext retrieval, but remains below the state of the art, especially for English–Chinese retrieval.The comparison uses precision@1 against results from Yang et al. (2019).
- Retrieval Question Answering (ReQA): SQuAD is converted into ReQA by treating questions as queries and retrieving the sentence containing each answer span.The experiments compare sentence- and paragraph-level retrieval, including comparisons with BM25 for paragraphs.
- Bitext Retrieval (BR): Performance below Yang et al. (2019) is attributed to one SentencePiece vocabulary covering 16 languages, which can split some languages into single characters.The authors identify this effect particularly for Chinese, Korean, and Japanese.
- Cross-lingual Retrieval: Cross-lingual SR and ReQA machine-translate queries while retaining English candidates; USETrans outperforms USECNN across languages, with performance surprisingly close to English-only tasks.Paragraph retrieval with the model first retrieves the nearest sentence and then selects its enclosing paragraph.
6 Experiments on Transfer Tasks
English transfer performance is evaluated on SentEval using task-specific downstream networks and embedding-based similarity. The multilingual models are competitive with state-of-the-art sentence encoders, with USETrans outperforming USECNN and sometimes exceeding the prior English-only USETrans model.
- Evaluation: English transfer performance is evaluated on SentEval, using task-specific DNNs for classification and embedding similarity for pairwise semantic similarity.Pairwise similarity is assessed from the sentence embeddings of u and v.
- Results: USETrans outperforms USECNN on all transfer tasks and shows competitive performance with state-of-the-art sentence embedding models.The comparison includes prior English-only sentence embedding models.
- Results: The multilingual USETrans model exceeds the previously released English-only USETrans model on some English transfer tasks.This result is reported as a comparison across the evaluated transfer tasks rather than as a universal improvement.
7 Resource Usage
The benchmarks compare compute and memory usage for the multilingual Transformer and CNN models, including effects of hardware and sentence length. CNN models are fastest and have the smallest memory footprint, but this efficiency comes with lower accuracy on many tasks.
- CNN models have the smallest memory footprint and are fastest on both CPU and GPU.
- Transformer memory requirements increase more than twice as fast as CNN requirements as sentence length grows.
- The efficiency advantage of CNN models for longer texts corresponds to a drop in accuracy on many retrieval and transfer tasks.
8 Conclusion
The paper presents multilingual sentence encoders that embed 16 languages into a shared space and perform well across retrieval and transfer tasks. Cross-lingual retrieval performance approaches monolingual performance for many language pairs, while Transformer models have higher resource costs as sequences grow.
- The models embed text from 16 languages into a shared semantic embedding space.
- The models achieve good performance on semantic retrieval, bitext retrieval, and retrieval question answering.
- Transformer time and space complexity is ultimately O(n^2), although shorter sequences are dominated by linear-scaling computations with larger constant factors.
- Cross-lingual semantic retrieval and retrieval question answering approach monolingual performance for many language pairs.