Keyword Assisted Topic Models

1 Introduction

• Automated topic models may create multiple topics with similar content or combine distinct themes, complicating clear interpretation and measurement.Because these models do not directly incorporate information about topics of interest, researchers may not know whether estimated topics measure those concepts until after fitting.Source: §1 · p. 3
• Post-hoc labeling and topic adjustments can compromise the objectivity of empirical findings.Researchers must connect estimated topics to substantive concepts after fitting, including through decisions such as merging topics or reweighting words.Source: §1 · pp. 3–4
• keyATM uses keywords specified before fitting to label topics and measure substantive concepts, rather than relying on post-hoc interpretation and adjustment.The model is semi-supervised, combining a small amount of keyword information with unlabeled data, without requiring manual labeling of many documents.Source: §1 · p. 4
• The authors report that a small number of keywords substantially improves topic-model performance and better serves measurement.They assess performance qualitatively and through crowd-sourced validation, and quantitatively by comparing document classification with human coding.Source: §1 · p. 4
• keyATM permits topics without keywords, allows documents to belong to multiple keyword topics, and avoids strict keyword constraints and iterative refinement.The paper also distinguishes keyATM from a related model that assumes one keyword topic per document.Source: §1 · p. 5
• keyATM can incorporate document-level covariates and time trends, and its fully Bayesian approach estimates hyperparameters to improve model performance.The paper presents the method as particularly useful for measuring specific topics; for exploration, existing models such as LDA may be more appropriate.Source: §1 · pp. 4–5

2 The Base keyATM

• The base keyATM extends Jagarlamudi, Daum´e III and Udupa (2012) by allowing topics without keywords and estimating hyperparameters for empirical performance.Source: §2 · p. 6

2.1 Model

• For keyword topics, each token is drawn either from the supplied keyword set or the ordinary topic-word distribution, with a probability governing keyword use.Source: §2.1 · pp. 7–8

2.2 Sampling Algorithm

• A collapsed Gibbs sampler integrates out (θ, φ, eφ, π), while wLDA term weights limit frequent words from overwhelming meaningful words.Source: §2.2 · pp. 9–10

2.3 Model Interpretation

• Topic-word and document-topic distributions characterize topic content and document themes, respectively, and are estimated using marginal posterior means.Source: §2.3 · p. 11

2.4 Empirical Evaluation

• keyATM produces more interpretable topic-word distributions than wLDA, aligns them more closely with CAP labels, and outperforms wLDA against CBP human coding for bill classification.Source: §2.4 · p. 12

2.4.1 Data and Setup

• The evaluation uses 4,421 bills and 21 CAP topics, comparing keyATM with otherwise matched keyword-free wLDA; reported findings are robust to keyword selection.Source: §2.4.1 · pp. 12–14

2.4.2 Topic Interpretability

• For Labor, keyATM’s top words include relevant terms such as “benefit,” “employee,” and “compensation,” while wLDA’s contain unrelated terms.Source: §2.4.2 · p. 14, Table 2 · p. 15
• For Foreign trade, wLDA lacks meaningful terms and splits related terms across Transportation and Foreign trade, whereas keyATM’s top words represent both topics.Source: §2.4.2 · pp. 14–15, Table 2 · p. 15
• keyATM separates Law & crime from Immigration, while wLDA struggles to represent Law & crime and conflates its keywords with Immigration.Immigration bills account for only 1.18% of all bills.Source: Table 2 · p. 15, §2.4.2 · pp. 15–16
• Both keyATM and wLDA fail to identify meaningful words for Government operations, the corpus’s most frequent topic.Source: Table 2 · p. 15, §2.4.2 · p. 16

2.4.3 Topic Classification

• Across six selected topics, keyATM classifies better than wLDA except for Government operations, and its ROC results are less sensitive to starting values.The comparison uses topic-document distributions against human coding; the reported exception and starting-value pattern are shown in Figure 1.Source: §2.4.3 · pp. 16–17, Figure 1 · p. 17

3 The Covariate keyATM

• The covariate keyATM models document-topic distributions using document covariates and a Dirichlet-Multinomial regression framework compatible with its collapsed Gibbs sampler.The covariates can include document metadata such as authorship.Source: §3 · p. 17

3.2 Empirical Evaluation

• Covariate keyATM yields more interpretable topics and results less sensitive to the total number of topics than STM.The authors characterize its results as more reasonable and robust than STM's.Source: §3.2 · p. 18, §3.2.5 · p. 24

3.2.1 Data and Setup

• The analysis covers a total of 7,497 manifestos, spanning eight consecutive elections and 3,303 unique candidates who ran in those elections.It uses the document-term matrix from the original study.Source: §3.2.1 · pp. 18–19

3 The Covariate keyATM

• The authors do not directly model correlations across topics, though they report that the proposed models generally perform well without doing so.They explored alternative modeling strategies, including a logistic-normal approach.Source: §3.2.1 · p. 18

3.2.2 Keyword Construction

• Because the application lacks human-coded topics, the authors independently construct keywords from UTAS questionnaires rather than reuse words derived from the same manifestos.The resulting set has 16 topics: two pork-barrel and 14 programmatic-policy topics.Source: §3.2.2 · p. 19

3.2.3 Topic Interpretability

• Judged by the ten most frequent words, covariate keyATM produces more interpretable topics than STM, including relevant terms for road construction, education, and security.For education, STM omits “education,” while keyATM includes “children” and “education.”Source: §3.2.3 · pp. 20–21

3.2.4 Topic Discrimination

• KeyATM assigns the same keywords to fewer topics than STM, and more keywords appear among the most frequent terms for only one topic under keyATM.Figure 2 compares how often keywords occur among the top 10 and top 15 words.Source: §3.2.4 · p. 22, Figure 2 · p. 22

3.2.5 Covariate Effects

• After the 1994 electoral reform, programmatic-policy topic proportions increase while pork-barrel proportions remain virtually unchanged across keyATM specifications, except without no-keyword topics.STM's results are more sensitive to the number of topics, and its estimated change is less stark.Source: §3.2.5 · pp. 23–24

4 The Dynamic keyATM

• Dynamic keyATM incorporates time ordering to model topic-prevalence trends smoothly while accounting for uncertainty.It builds on the base keyATM's collapsed Gibbs sampler and applies a Hidden Markov Model.Source: §4 · p. 24

4.1 Model

• The model represents time periods with latent discrete states and permits only one-step forward transitions, an assumption that simplifies estimation when there are sufficiently many states.It allows topic proportions to evolve by varying α across latent states.Source: §4.1 · pp. 24–25

4.2 Empirical Evaluation

• In the Supreme Court opinion application, dynamic keyATM yields more interpretable topics and better classification performance than dynamic wLDA, with time trends closer to human coding.The comparison uses opinions with human-identified primary topics.Source: §4.2 · p. 25

4.2.1 Data and Setup

• The evaluation analyzes 17,245 Supreme Court opinions written between 1946 and 2012; SCD human coders identified primary issue areas, and its aggregate 14 categories serve as keyword topics.Source: §4.2.1 · p. 26

Appendix p. 72 for the full list of keywords).

• The evaluation uses all 14 SCD categories as keyword topics and includes no additional topics without keywords because the categories are treated as comprehensive.The comparison matches wLDA topics to SCD topics by maximizing classification performance; keyATM topics are labeled through their prespecified keywords.Source: Appendix p. 72 for the full list of… · p. 27

4.2.2 Topic Interpretability

• Dynamic keyATM's topics are at least as interpretable as wLDA's, but both models fail to identify meaningful terms for Privacy because its keywords rarely appear in assigned opinions.For the First Amendment topic, keyATM's top words include relevant terms such as “first,” “amendment,” and “speech.”Source: §4.2.2 · pp. 27–28

4.2.3 Topic Classification

• Dynamic keyATM outperforms wLDA in classification except for Privacy, and its ROC curves are generally less sensitive to starting values, also except for Privacy.Figure 4 compares ROC curves for six selected topics across the five chains.Source: Figure 4 · p. 29, §4.2.2 · p. 29, §4.2.3 · p. 30

4.2.4 Time Trends of Topic Prevalence

• Estimated topic-prevalence trends from keyATM correlate more highly with SCD human coding than wLDA for most topics, while both models correlate weakly for Privacy.The comparison standardizes the estimated and coded proportions to focus on relative time trends.Source: §4.2.4 · pp. 30–31, §4.2.3 · p. 30

5.1 The Validation Methodology

• The validation measures coherence by asking workers to identify an unrelated word set and measures label correspondence with a modified task that also presents a topic label.Responses are analyzed with Bayesian hierarchical logistic regressions, including worker- and topic-specific effects.Source: §5.1 · p. 31

5.2 Findings

• In pooled crowdsourced results, base keyATM performs better than wLDA in both validation tasks, with positive differences significant at the 95% level.Figure 5 plots differences in predicted probabilities of correct responses, with separate panels for pooled and topic-specific estimates.Source: §5.2 · pp. 32–34, Figure 5 · p. 33
• Covariate keyATM performs better than STM on coherency-and-label responses but similarly on topic coherence; dynamic keyATM and wLDA show no statistically meaningful performance difference.The authors suggest that technical vocabulary and complex concepts in court opinions may challenge crowdsourced evaluation.Source: §5.2 · p. 34
• Crowdsourced evaluation of Supreme Court opinions has a practical boundary: technical vocabulary and complex concepts can make the tasks challenging for workers.The authors give connecting the Civil rights label to words such as “district,” “school,” and “discrimination” as an example requiring background knowledge.Source: §5.2 · p. 34

6 Keyword Selection

• Poorly matched or infrequent keywords can impair keyATM’s topic interpretability and classification, making keyword quality central to its application.The Government operations keywords were less frequent in relevant bills than those for the other five topics.Source: §6 · p. 34
• keyATM outperforms wLDA even after some keywords are randomly removed, although keyword choice matters when the keyword set is extremely small.The Immigration topic performed poorly with only three keywords.Source: §6 · p. 34
• Researchers should examine candidate-keyword frequency and choose terms that substantively match their topics of interest.The authors refer readers to the appendix for the full set of results.Source: §6 · pp. 34–35
• The authors identify how to choose good keywords as an issue for future research.They cite automated keyword-selection algorithms and a dictionary-making procedure based on average frequency entropy as examples of related work.Source: §6 · p. 35

7 Concluding Remarks

• Providing a small number of keywords can improve topic interpretability and classification performance, while keyATM requires researchers to label topics before fitting.The conclusion also describes documents as able to belong to multiple topics.Source: §7 · p. 35

A Graphical Representation of keyATM

• The base, covariate, and dynamic keyATM graphical models distinguish observed variables from latent variables.The base and dynamic diagrams mark w as observed; the covariate diagram marks w and x as observed.Source: Figure S1 · p. 42, Figure S2 · p. 43, Figure S3 · p. 43

C.1.1 Sampling Algorithm

• The covariate sampler modifies the base sampler by drawing word-topic assignments from a conditional posterior that incorporates covariates.The model uses standardized covariates during estimation and can rescale coefficients for raw covariates.Source: §C.1.1 · pp. 44–45

C.1.2 Model Interpretation

• Covariate keyATM does not directly model topic-word distributions, but these can be examined across covariate values using documents that share those values.Source: §C.1.2 · p. 46

C.2.1 Sampling Algorithm

• The dynamic sampler adds latent-state and transition-probability sampling, using forward-and-backward sampling for state membership.The procedure samples states backward from hT, with hT = R and h1 = 1.Source: §C.2.1 · pp. 46–47

C.2.2 Model Interpretation

• Dynamic topic prevalence is estimated from document-level posterior topic proportions averaged across documents at each time point.The resulting quantity can be plotted to visualize topic prevalence trends.Source: §C.2.2 · p. 47

C.3 Keywords as Informative Prior

• keyATM restricts its prior structure by assigning no-keyword frequencies exactly zero, allowing the data to inform keyword importance without researchers specifying hyperparameter values.Source: §C.3 · p. 48

Documents; CAP Codebook

• The applications use shared preprocessing across models, while CAP keyword selection removes irrelevant phrases, limits unjustified reuse, and caps each topic at 25 keywords.The preprocessing removes stop words and rare terms and lemmatizes remaining terms.Source: Documents · p. 48, CAP Codebook · p. 48

D.3 Top Words

• Table S2 compares each model’s ten highest-probability words by topic, and keyATM performs better than wLDA in most topics.The table marks keyATM’s prespecified keywords in bold and keywords specified for another topic with asterisks.Source: §D.3 · p. 50, Table S2 · p. 52

D.4 ROC Curves

• keyATM performs at least as well as wLDA in all topics on the ROC comparison across five chains.The figure shows the median AUROC among five chains for each model.Source: §D.4 · p. 52

D.6 Quality of Keywords and Performance of the keyATM

• Government operations keywords occur less often in relevant bills and have fewer unique terms than keywords for the other five topics, a pattern linked to weaker interpretability and classification.The authors conclude that high-quality keyword selection is critical for successful keyATM application.Source: §D.6 · pp. 52–53

D.5 Pooled Results

• In pooled results, keyATM substantially outperforms wLDA in AUROC.Figure S5 pools the results from five Markov chains for each model.Source: §D.5 · p. 52, Figure S5 · p. 55

D.7 The Results based on Different Keywords

• The alternative keyword set retains overlapping and substantively peripheral terms, making it much larger than the main-analysis set.Source: §D.7 · p. 57

D.7.1 The Full List of Different Keywords

• Table S3 documents the alternative keyword set used for the base keyATM application.Source: Table S3 · p. 60

D.7.2 Top words

• With different keywords, keyATM yields related Labor words, trade terms for Foreign trade, and separate Law & crime and Immigration topics, unlike wLDA.wLDA’s Foreign trade top words instead resemble Transportation, while its Labor topic lacks related terms.Source: §D.7.2 · p. 60, Table S4 · p. 62

D.7.3 ROC Curves

• keyATM performs at least as well as wLDA in 19 of 21 topics across the five chains.The median AUROC is computed across five chains for each model.Source: §D.7.3 · p. 62

D.8 Randomly Removed Keywords

• Even after randomly removing 50% of keywords, keyATM’s worst AUROC is above wLDA’s in most cases; Immigration is a notable exception with high variability.The Immigration keyword sets contain only two of the original three terms.Source: §D.8 · p. 63, Table S5 · p. 63
• Immigration classification performs substantially worse when “immigration” is absent, and poor-performing sampled sets all omit that term.This topic had only three original keywords: “citizenship”, “immigration”, and “refugee”.Source: §D.8 · pp. 63–64

D.9 Convergence

• The Gelman-Rubin statistic for perplexity decreases monotonically and reaches 1.08 after 300,000 iterations, with the first third dropped as burn-in.Source: §D.9 · p. 64

E.1 Background

• The Japan application concerns a mixed-member majoritarian electoral system introduced in 1994 and first used in the 1996 election.The account notes that the reform changed LDP candidates’ electoral strategies toward programmatic policies rather than pork barrel.Source: §E.1 · p. 68

E.2 Keyword Construction Process

• For the Japan application, the authors independently map Catalinac’s topic labels to UTAS policy questions, remove five unmatched areas, then select nouns as keywords.The resulting keywords avoid reusing wLDA’s most frequent output words; the UTAS surveys have an average response rate of approximately 85%.Source: §E.2 · p. 68

E.3 Top Words

• The Japan application presents keyATM and STM top words side by side for policy areas including Public Works, Regional devolution, and Tax.Table S9 reports the ten highest-probability words for each topic and marks keyATM’s pre-specified keywords.Source: §E.3 · p. 72, Table S9 · p. 74

F.1 Introducing Time Dynamics with HMM

• Unlike some HMM-based dynamic topic models, dynamic keyATM lets multiple documents share a state, allows multiple topics per document, and uses keywords.The passage contrasts it with approaches that restrict documents to one topic or do not use keywords.Source: §F.1 · p. 74

F.2 Keyword Construction Process

• For the dynamic application, keywords come from Supreme Court Database issue descriptions, are lemmatized and filtered, and are limited to 25 per topic.Repeated terms across topics are retained only when substantively justified; excess terms are removed based on their corpus proportion.Source: §F.2 · pp. 74–75

F.3 The Full List of Keywords

• Table S10 lists the keywords used in the dynamic keyATM application.Source: Table S10 · p. 75

F.4 Top Words

• The authors report that keyATM performs better than wLDA in most topics in the top-word comparison.Table S11 presents the ten highest-probability words for each topic and model.Source: §F.4 · p. 76, Table S11 · p. 77

F.5 ROC Curves

• keyATM performs at least as well as wLDA in 11 of 14 topics by median AUROC across five chains.The ROC curves show results from five chains with different starting values for each model.Source: §F.5 · p. 77, Figure S11 · p. 78

F.6 Correlation of topic prevalence

• keyATM’s estimated topic prevalence correlates more highly with SCD human coding than wLDA for 9 of 13 topics.Private action is omitted because all associated documents are from 2012; the table uses standardized measures.Source: §F.6 · p. 77, Table S12 · p. 79

F.7 Quality of Keywords

• Privacy keywords appear less often and in fewer unique forms in SCD-coded Privacy opinions than keywords for the other five topics.The authors suggest careful keyword selection because the quality of Privacy keywords seems related to poor interpretability and classification.Source: §F.7 · p. 78, Figure S12 · p. 79

G Additional Information for the Topic Matching of wLDA

• For wLDA, the authors match topic labels to model topics with the Hungarian algorithm, maximizing AUROC and using a harmonic mean across labels.keyATM instead uses pre-specified topic keywords without post-hoc labeling.Source: G Additional Information for the Topic… · pp. 79–80

H.1 Design

• Validation participants complete five tasks for each model and one gold-standard task; responses are excluded if they fail any gold-standard item.Tasks and answer options are randomized, and the T8WSI trial had more gold-standard failures than the coherency tasks.Source: §H.1 · p. 80

H.2 Alternative design

• The validation omits Word Intrusion and T8WSI because these tasks are difficult, with T8WSI especially sensitive to displayed-word choice.In the trial, 18.7% failed the T8WSI gold-standard task, versus 6.7% for coherency and 8.0% for coherency-and-label.Source: §H.2 · pp. 81–82

H.3 Descriptive Statistics

• Each topic has about 20–30 validation tasks, with the exact number varying because tasks were randomly selected.Figures 5 and S15 report pooled and topic-by-topic results.Source: §H.3 · p. 82

• Economic activity is one weaker keyATM topic: its AUROC is lower, and its top words are less interpretable than wLDA’s.The passage reports that Table S11 gives wLDA more meaningful words for this topic.Source: §H.4 · p. 82
• For legislative bills, the base-model tables report correct and total task counts for coherency and coherency-and-label validation.Source: Table S15 · p. 83, Table S16 · p. 84
• For manifestos, the covariate-model tables report correct and total task counts for coherency and coherency-and-label validation.Source: Table S17 · p. 85, Table S18 · p. 85
• For court opinions, the dynamic-model tables report correct and total task counts for coherency and coherency-and-label validation.Source: Table S19 · p. 86, Table S20 · p. 86
• keyATM performs better than wLDA in most topics, but performs worse for some topics in the Supreme Court opinion application.Figure S15 compares predicted-probability differences across validation tasks.Source: §H.4 · p. 82, Figure S15 · p. 87
• Figure S15 plots predicted-probability differences, with black and blue lines distinguishing coherency from coherency-and-label tasks and intervals showing 95% and 90% credible intervals.Source: Figure S15 · p. 87