Source-linked AI summary
Multilingual Topic Models for Unaligned Text
Jordan Boyd-Graber, David Blei
TL;DR
Topic models typically assume monolingual corpora, limiting analysis of multilingual collections and corpus exploration across languages. MUTO jointly learns multilingual topic spaces and cross-language matchings, recovering matched documents better than the prior alone and supporting broader multilingual applications.
Problem
Monolingual topic-model assumptions limit opportunities for analyzing multilingual corpora and helping readers navigate documents across languages.
Method
MUTO simultaneously finds topic spaces and matchings across multiple languages, using matching regularization and topic structure during inference.
Results
MUTO recovers matched documents better than the prior alone; on Wikipedia, all MUTO methods outperform the intersection baseline, while the union baseline is little better than random guessing.
Takeaways & Limitations
MUTO offers a foundation for multilingual topic-model applications and corpus exploration, with possible extensions to relationships between words and other discrete vocabularies or modalities.
Takeaways & Limitations
Poor matches can dilute topics by reinforcing pairings that combine different themes, such as mathematics and Roman history.
Abstract
from arXiv · showhide
We develop the multilingual topic model for unaligned text (MuTo), a probabilistic model of text that is designed to analyze corpora composed of documents in two languages. From these documents, MuTo uses stochastic EM to simultaneously discover both a matching between the languages and multilingual latent topics. We demonstrate that MuTo is able to find shared topics on real-world multilingual corpora, successfully pairing related documents across languages. MuTo provides a new framework for creating multilingual topic models without needing carefully curated parallel corpora and allows applications built using the topic model formalism to be applied to a much wider class of corpora.
1 Model
MUTO connects vocabulary terms across languages with a latent matching and uses matched pairs to form multilingual topics, while unmatched terms come from language-specific background distributions.
- Matching across Vocabularies: MUTO assumes similar themes across languages and discovers vocabulary-level parallelism without explicit parallel documents.The model can use cross-language thematic correspondence to infer likely word matches.
- Matching across Vocabularies: The matching is a bipartite graph linking each term to at most one term in the opposite language, with edge preferences encoded by π.The prior can incorporate morphological features, dictionaries, or information induced from non-parallel text.
- From Matchings to Topics: MUTO defines topics as distributions over matched word pairs rather than individual vocabulary terms.For example, a matched pair such as (“hund”, “hound”) can receive high probability in a pet-related topic.
- From Matchings to Topics: Each document has topic weights, and each word is generated either from a topic-specific matched pair or a language-specific unmatched-word distribution.The matching indicator determines whether the observed language-consistent member of a pair or a background term is selected.
- From Matchings to Topics: The background distribution is shared across documents and prevents unmatched frequent nouns from becoming divergent language-specific topic content.This design also encourages words well explained by a language-specific background model to remain unmatched.
2 Inference
MUTO uses stochastic EM to alternate between sampling topic assignments and improving the vocabulary matching, with matching weights derived from topic and background likelihoods.
- Inference: Stochastic EM alternates between inferring topic assignments given a matching and finding a maximum a posteriori matching given topic assignments.The procedure combines collapsed Gibbs sampling with matching optimization.
- Topic assignment: Collapsed Gibbs sampling assigns matched words using document-topic and pair-topic counts, while unmatched words use document topic assignments alone.For a matched pair, counts from both language-specific terms contribute to the pair’s topic count.
- Matching optimization: The Hungarian algorithm selects the matching by maximizing edge weights computed from posterior topic and background estimates.Adding an edge replaces separate language-background contributions with topic-specific contributions for the paired terms.
- Matching optimization: Matching favors words that share topics, are poorly explained by background language models, and satisfy prior preferences.Specialized-context words are therefore better modeled by topic distributions than by background distributions.
- Initialization: MUTO initializes the matching with shared words longer than five characters, which is intended to provide a high-precision seed for similar-orthography languages.This initialization is subsequently improved during inference.
- Overfitting control: To limit overfitting, inference stops after three M steps with 250 Gibbs iterations per stochastic E step, while the allowed matching size grows gradually.More principled control of matching complexity or richer matching priors is left for future work.
3 Data
The evaluation uses bilingual corpora and several sources of matching priors, including parallel-corpus statistics, dictionaries, edit distance, and MCCA-derived weights.
- Evaluation resources: MUTO is evaluated on two corpora using four sources of matching prior information, selected according to available resources for the language pair.The priors encode preferences over candidate cross-language matches.
- Matching priors: Parallel-corpus PMI provides matching priors even when the target corpus itself is not parallel.PMI is computed for terms appearing in translations of aligned sentences in a small German-English news corpus.
- Matching priors: A machine-readable dictionary assigns prior weight 1/N to each of N possible translations and effectively disallows dictionary-excluded links.This gives additional weight to unambiguous translations.
- Matching priors: Edit-distance priors use string similarity when reliable linguistic resources are unavailable, with π_i,j = 1/(0.1 + ED(v_i, v_j)).The experiments penalize all substitutions and deletions equally.
- Matching priors: MCCA supplies another prior by learning a mapping from term feature vectors to latent points and using latent-space distances for matching weights.MCCA is run on the same bilingual dataset before its weights are used as MUTO priors.
- Corpora: The experiments use parallel corpora for evaluation while withholding their document alignment from MUTO, preserving the model’s non-parallel training setting.Parallel structure guarantees the shared-theme assumption needed for evaluation.
- Corpora: The datasets include 2,796 directly translated German-English Europarl documents and cross-linked German-English Wikipedia articles with more variable content.Wikipedia articles can cover different aspects of the same topic across languages.
- Preprocessing: The vocabulary is restricted to nouns, using the 2,500 most frequent terms to retain topic-relevant words and make matching computation tractable.Removing nonnouns also excludes frequent but less informative inflected verbs.
4 Experiments
The experiments evaluate MUTO’s learned topics, vocabulary matchings, and cross-language document retrieval. Matching priors support coherent topics and accurate vocabulary connections, while data-adapted matchings can improve document matching but may fail when linguistic cues mismatch corpus structure.
- 4.1 Learned Topics: MUTO learns thematically coherent bilingual topics represented as distributions over paired terms.Wikipedia topics include computers and science, with pairs such as “computer,” “lovelace,” and “software” identified through edit-distance guidance.
- 4.1 Learned Topics: Without an appropriate matching prior, spurious matches accumulate and pull topics in conflicting directions, producing incoherent topics and incorrect matches.The no-prior Wikipedia model shows poor translations accumulating over iterations.
- 4.3 Matching Documents: For Europarl, the intersection-vocabulary baseline performed well, whereas on Wikipedia it underperformed all MUTO methods; the union baseline was little better than random on both corpora.The intersection baseline benefited from the truly parallel structure of Europarl.
- 4.3 Matching Documents: Morphological cues can find high-accuracy vocabulary matchings without reliably recovering parallel documents across corpora.Morphological methods performed little better than chance on homogeneous Europarl, while the edit-weight prior worked well on Wikipedia.
- 4.2 Matching Translation Accuracy: Matching priors alone outperform LDA baselines and MUTO without a prior on Wikipedia, while updating matchings with data improves document matching in many conditions.For vocabulary translation accuracy, adding topic structure neither hurts nor helps relative to prior weights alone.
5 Discussion
MUTO jointly learns multilingual topic spaces and document matchings, supporting exploration of multilingual corpora without requiring monolingual or carefully curated parallel-text assumptions. Its broader symbol-level formulation also suggests applications beyond bilingual text, while incorporating local syntax could improve translation quality.
- MUTO simultaneously finds topic spaces and matchings in multiple languages, recovering matched documents better than the matching prior alone on real-world data.
- Multilingual topic models could help users locate relevant documents in a language of interest by providing readable signposts in a more familiar language.
- MUTO makes no linguistic assumptions that preclude learning relationships between symbols from other discrete vocabularies.
- The same relationship-learning framework could help explain and annotate pairings such as words with images, sound, genes across organisms, or metadata with text.
- Incorporating local syntax into MUTO, rather than using it only as a matching prior, could improve translation quality without parallel corpora.
- MUTO removes the monolingual-corpus assumption from topic models and opens opportunities for multilingual applications in machine translation, linguistic phylogeny, and semantics.