Source-linked AI summary
Cross-lingual Contextualized Topic Models with Zero-shot Learning
Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, Elisabetta Fersini
TL;DR
Traditional topic models rely on language-specific bag-of-words representations, limiting transfer across languages and complicating multilingual vocabularies. This paper introduces a zero-shot cross-lingual neural topic model that replaces BoW inputs with multilingual contextualized representations, learns topics in one language, and applies them to unseen languages. The resulting model supports cross-lingual topic modeling, with transferred topics evaluated for coherence and stability across languages.
Problem
Traditional topic models depend on language-specific bag-of-words vocabularies, limiting application to other languages and creating oversized, sparse multilingual vocabularies.
Method
The model replaces BoW document inputs with multilingual contextualized embeddings, trains on one language, and predicts topics for unseen documents in other languages.
Results
The resulting model enables zero-shot cross-lingual topic modeling and can be trained on one language and applied to another language with available embeddings.
Takeaways & Limitations
Contextualized embeddings can replace input BoW representations while supporting zero-shot topic modeling across languages.
Takeaways & Limitations
The approach depends on a multilingual embedding model covering the target language.
Abstract
from arXiv · showhide
Many data sets (e.g., reviews, forums, news, etc.) exist parallelly in multiple languages. They all cover the same content, but the linguistic differences make it impossible to use traditional, bag-of-word-based topic models. Models have to be either single-language or suffer from a huge, but extremely sparse vocabulary. Both issues can be addressed by transfer learning. In this paper, we introduce a zero-shot cross-lingual topic model. Our model learns topics on one language (here, English), and predicts them for unseen documents in different languages (here, Italian, French, German, and Portuguese). We evaluate the quality of the topic predictions for the same document in different languages. Our results show that the transferred topics are coherent and stable across languages, which suggests exciting future research directions.
1 Introduction
Traditional topic models depend on language-specific bag-of-words vocabularies, limiting transfer across languages and creating sparse, unwieldy multilingual vocabularies. The paper replaces BoW inputs with multilingual contextualized representations to enable zero-shot cross-lingual topic modeling.
- Traditional topic models use fixed, language-specific vocabularies and cannot readily transfer to other languages.
- Training jointly on several languages produces vast vocabularies that create parameter-size, search, and overfitting problems.
- Zero-shot learning trains on one language and tests on languages unavailable during training, assuming topics share the same gist across languages.
- The proposed architecture replaces document BoW representations with multilingual contextualized embeddings.
- The model addresses unseen words and unseen-language documents without requiring test-language BoW representations, and evaluates topics in four test languages.
- The model is presented as a novel neural topic model using language-independent representations to generate topic distributions and enabling zero-shot cross-lingual tasks.
2 Contextualized Neural Topic Models
The model extends a neural variational topic-modeling framework by replacing its BoW input with multilingual contextualized representations. BoW remains useful for training topic indicators, while language-independent representations support zero-shot modeling in unseen languages.
- The model extends Neural-ProdLDA, a neural topic model based on a Variational AutoEncoder.
- Pre-trained multilingual SBERT representations replace Neural-ProdLDA’s input BoW, while the reconstructed BoW layer remains for training topic indicators.
- The reconstructed BoW becomes unnecessary during testing because topic indicators are obtained during training.
- ZeroShotTM uses word-order and contextual information to overcome a central BoW limitation and model unseen languages.
- The approach supports low-resource cross-lingual settings when multilingual contextualized embeddings cover the target language.
3 Experiments
The experiments test whether ZeroShotTM can replace bag-of-words inputs while preserving topic coherence and support zero-shot prediction across unseen languages. Evaluations compare coherence, cross-lingual topic similarity, automatic baselines, and human judgments.
- Experimental goals: Experiments test contextualized representations against bag-of-words topic models and evaluate zero-shot cross-lingual topic modeling.The study also documents datasets, implementation, runtime, and replication details in the Appendix.
- Datasets: W2 contains 100,000 English documents; 99,700 are used for training, while 300 comparable documents are collected in Portuguese, Italian, French, and German.The multilingual test instances refer to the same Wikipedia entities across languages.
- Monolingual evaluation: ZeroShotTM obtains comparable NPMI coherence to Combined TM and Neural-ProdLDA on W1, showing contextualized embeddings can replace BoW inputs without loss of coherence.Results are averaged over 30 runs for 50 and 100 topics.
- Quantitative evaluation: Cross-lingual evaluation compares matches, centroid similarity, and KL divergence between each unseen-language document and its English counterpart.Higher match and centroid similarity are better, while lower KL divergence indicates more similar topic distributions.
- Quantitative evaluation: More than 70% of predicted topics match the corresponding English topics, outperforming uniform baselines and remaining in line with translated-document results.Centroid similarity indicates that nonmatching predictions can still be similar to the English topic, without translation.
- Qualitative evaluation: Manual evaluation rates predicted topics for 300 test documents in five languages, while examples show the model predicts the main topic in languages unseen during training.Human scoring reliability is 0.88 using Gwet AC1 with ordinal weighting.
4 Related Work
Prior multilingual topic models use parallel corpora, dictionaries, or other language-dependent resources, whereas zero-shot prediction for unseen languages remains unsupported.
- Existing multilingual and cross-lingual topic models require multilingual training corpora and remain language-dependent.
- The Polylingual Topic Model applies LDA to parallel corpora, assuming translations share the same topic distributions.
- Other approaches transfer knowledge at the document level or use multilingual dictionaries requiring predefined mappings.
- Word and document embeddings capture semantic, syntactic, and social aspects of language, motivating language-independent topics.
5 Conclusions
The paper proposes contextualized document embeddings as input for cross-lingual topic modeling. Its results support replacing input BoW representations and applying the trained model to other languages with available embeddings.
- The paper proposes a neural architecture for cross-lingual topic modeling using contextualized document embeddings as input.
- Contextualized embeddings can replace the input BoW representations.
- Using contextualized representations enables zero-shot cross-lingual topic modeling.
- The resulting model can be trained on one language and applied to another language for which embeddings are available.
A Datasets
The datasets were built from English DBpedia abstracts, including a 20,000-document collection and a multilingual set derived from 300 retrieved abstract translations.
- The datasets were created from the English DBpedia 2016-10 abstract dump.
- W1: W1 contains 20,000 randomly sampled English documents for quickly testing whether Contextual TM performance decreases significantly.
- W2: W2 begins with 100,000 abstracts of at least 200 characters, from which 300 random English abstracts were selected.
- W2: For the 300 selected abstracts, corresponding versions in other languages were retrieved through the DBpedia SPARQL endpoint.
- W2: Manual evaluation of the 300 retrieved abstracts found no mismatch or corrupted text.
A.1 Preprocessing
The preprocessing pipeline removes punctuation, digits, and English stop-words, then limits each abstract’s BoW to its 2,000 most frequent words.
- The documents undergo a standard preprocessing pipeline before modeling.
- The pipeline removes punctuation, digits, and NLTK’s English stop-words.
- Each abstract retains only its 2,000 most frequent words for the BoW representation.
B.1 Neural-ProdLDA
The Neural-ProdLDA baseline uses a standard neural topic-model implementation with specified training settings and compares related variants using BoW and SBERT representations.
- The implementation uses 100 training epochs, ADAM optimization, a single hidden layer with 100-dimensional softplus units, and learnable topic and document priors.Momentum is 0.99, the learning rate is 0.002, dropout is 20%, and batch size is 200.
- The Neural-ProdLDA model uses the same hyperparameters as the SBERT-only variant, except Neural-ProdLDA replaces SBERT features with BoW inputs.
- The combined model concatenates SBERT embeddings transformed to vocabulary size with the BoW representation.The transformation is implemented as a learnable dense layer from R512 to R|V|.
- The LDA implementation estimates alpha and beta from the data during training.Alpha controls the document-topic distribution, while beta controls the word-topic distribution.
- Experiments ran on two laptops with GeForce GTX 1050 GPUs, CUDA 10, and 16GB of RAM, and could also be replicated on CPU.
C.1 Runtime
ZeroShotTM has comparable runtime on common hardware, although Neural-ProdLDA is slightly faster because it avoids the additional dense representation.
- Neural-ProdLDA is slightly faster than ZeroShotTM because ZeroShotTM includes an additional representation that cannot be encoded as a sparse matrix.
- The authors characterize the runtimes as comparable and describe the model as usable on common hardware.
- Table 5 reports the time to complete one epoch on W1 with 25 and 50 topics.
- The software is available as an installable Python package.