Neural Models for Documents with Metadata

Introduction

• A general neural framework based on variational methods supports approximate Bayesian inference without model-specific derivations.The authors identify custom inference algorithms as a barrier to wider practical use of LDA variants.Source: §1 · p. 1
• SCHOLAR combines supervised LDA and SAGE-style customization, distinguishing metadata used as predictive labels from observed covariates that affect words.Options include covariate and topic-interaction effects, label supervision, and a rich encoder using word embeddings or interpretable covariate embeddings.Source: §1 · pp. 1–2
• Sparse deviations from a background log-frequency can be used to encourage more interpretable topics.The paper presents sparsity as an optional customization.Source: §1 · p. 2
• SCHOLAR trades off perplexity, coherence, and sparsity, improves coherence with word vectors, and supports an exploration of US immigration news.The paper also describes learning topics predictive of labels or explicit deviations for parts of the metadata.Source: §1 · p. 2
• The authors frame the VAE inference approach as a flexible modeling framework with potential for further extensions.They emphasize adapting the model to data characteristics.Source: §1 · p. 2

Background and Motivation

• LDA factors document-word frequencies into document-topic and topic-word matrices, with each row representing a distribution over topics or words, respectively.The probabilistic account uses latent Poisson mean parameters for the observed count matrix.Source: §2 · p. 2
• LDA topics can be coherent and interpretable, but without additional prior knowledge they may not reflect metadata of research interest, such as sentiment, ideology, or time.The document-topic representation can also serve as a measurement for downstream tasks.Source: §2 · p. 2
• SLDA models labels generated conditional on topics, while SAGE represents words with sparse deviations from background frequency and can add covariate and topic interactions.SLDA can support inductive text classification; SAGE constrains document-topic representations to the simplex.Source: §2 · pp. 2–3
• LDA variants often require custom inference derivations; SAGE breaks conjugacy and limits covariates to a single categorical label for computational efficiency.In SLDA, changing the assumed label distribution can require modifying the inference algorithm.Source: §2 · p. 3
• SCHOLAR adapts variational-autoencoder inference to combine ideas from SAGE and SLDA, allowing model modifications without changing the inference procedure.The paper identifies gradual topic evolution and hierarchical models as possible extensions.Source: §2 · p. 3

3 SCHOLAR: A Neural Topic Model with Covariates, Supervision, and Sparsity

• The framework leaves metadata-informed document priors, such as Dirichlet-multinomial regression, as a possible future combination.The authors hypothesize this approach could be combined with SCHOLAR.Source: §3 · p. 3

3.1 Generative Story

• SCHOLAR maps a logistic-normal latent document representation to words and optional labels, with covariates and topic–covariate interactions contributing deviations from a background frequency term.The model can omit labels, covariates, interactions, or sparsity depending on the data and modeling goals.Source: §3.1 · pp. 3–4

3.2 Learning and Inference

• A neural encoder approximates each document’s latent posterior, and reparameterized sampling supports stochastic ELBO optimization; this inference approach is not restricted to covariates with few unique values.The encoder uses document words and can incorporate covariates and labels.Source: §3.2 · pp. 4–5

3.3 Prediction on Held-out Data

• At test time, one encoder pass estimates posteriors; covariate-based label search suits small categorical sets, not many labels, multiclass classification, or regression.The encoder can also be retrained without labels for label prediction.Source: §3.3 · pp. 5–6

3.4 Additional Prior Information

• Fixed word-vector initialization can add unlabeled-text similarity information and may promote topic coherence; generating embeddings instead of tokens remains future work.The framework can also learn covariate embeddings that may represent metadata.Source: §3.4 · p. 6

4 Experiments and Results

• The evaluation reports held-out perplexity, NPMI topic coherence, and test accuracy; the immigration corpus contains approximately 4,000 timestamped articles annotated for pro- or anti-immigration tone.Coherence is measured using each topic’s top 10 words, internally on test data and externally on English Gigaword; reported perplexity is an ELBO-based upper bound.Source: §4 · p. 6

4.1 Unsupervised Evaluation

• SCHOLAR produces more coherent topics than LDA but has worse perplexity; SAGE gains sparsity at the cost of worse perplexity and coherence, while NVDM has relatively low perplexity but poor coherence.The results are for unsupervised evaluation; Table 1 covers IMDB with a 5,000-word vocabulary and 50 topics.Source: §4.1 · p. 7, Table 1 · p. 7
• More training epochs tend to improve held-out perplexity but worsen coherence, while sparse-topic regularization worsens both measures and creates sparse topics.Fixed pretrained word2vec embeddings gave the best internal coherence on IMDB and Yahoo Answers, and the second-best on 20 Newsgroups.Source: §4.1 · p. 7
• The background term has little effect on perplexity but helps produce coherent topics by accounting for common words, which otherwise receive greater weight in LDA topics.The comparison with ProdLDA is qualified because its batch-normalization layer may play a similar role and the implementations differ slightly.Source: §4.1 · p. 7

4.2 Text Classification

• SCHOLAR is competitive with logistic regression and outperforms SLDA in accuracy on three categorical-label datasets, while learning document representations relevant to the labels.The authors state that their aim is not state-of-the-art classification, but to demonstrate label-relevant representations with topic-model properties.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Exploratory Study

• Treating tone as a label yields coherent topics with face-valid pro-immigration probabilities; the topic “arrested charged charges agents operation” is least associated with pro-immigration tone.The topics are inferred jointly from words and tone, and Figure 2 displays the topic words and corresponding probabilities.Source: §4.3 · p. 8, Figure 2 · p. 8
• Treating tone as a covariate, including tone-topic interactions, yields word patterns that capture contrasting frames such as “criminal” versus “detainees” and “illegals” versus “newcomers.”Source: §4.3 · p. 8
• In the combined model, adjacent publication years receive similar learned deviations; one embedding dimension roughly tracks a shift from Clinton and INS to Obama and ICE, while the other reflects 9/11-related terms.The model treats tone as a label and year as a covariate, projecting year vectors into two dimensions.Source: §4.3 · p. 8, Figure 3 · p. 8

5 Additional Related Work

• Dirichlet-multinomial regression, Labeled LDA, MedLDA, and a neural extension of DMR are among other efforts to incorporate metadata into topic models.The neural DMR extension embeds metadata into a richer document prior.Source: §5 · p. 8
• Neural approaches have parameterized unsupervised document models, incorporated topics into a neural language model, and learned topic embeddings for a scalable correlated topic model.The cited work also includes non-Bayesian approaches.Source: §5 · pp. 8–9
• Table 3 presents topic words alongside anti-immigration and pro-immigration tone variations with interactions.Its columns place topic words at left and tone variations in the middle and right.Source: Table 3 · p. 9
• Other inference work extends reparameterization to Dirichlet and Gamma distributions, while general-purpose tools support black-box and VAE-style model exploration and evaluation.The cited extensions use transformations or generalized reparameterization.Source: §5 · p. 9

6 Conclusion

• SCHOLAR outperforms SLDA in predicting document labels.The conclusion reports this as a model result.Source: §6 · p. 9
• The model demonstrates a tradeoff between perplexity, coherence, and sparsity.The conclusion characterizes the tradeoff as manageable in the paper context.Source: §6 · p. 9
• Stochastic variational inference provides a general algorithm for framework variations without model-specific derivations.The paper presents this as a way to support flexible metadata incorporation.Source: §6 · p. 9
• The framework's flexibility supports exploration of a corpus of US immigration articles.The authors say their model and code will facilitate rapid exploration of document collections with metadata.Source: §6 · p. 9

A Deriving the ELBO

• For each document, the ELBO derivation integrates over latent r_i using q(r_i | w_i, c_i, y_i).The displayed bound begins from the joint p(w_i, y_i, r_i | c_i).Source: A Deriving the ELBO · p. 11
• The bound includes an expected log prior for r_i, word-likelihood terms, and an expected label-likelihood term.The expressions show log p(r_i), log p(w_ij | r_i, c_i), and log p(y_i | r_i, c_i).Source: A Deriving the ELBO · p. 11

B Model details

• The model-details section computes the KL divergence term with diagonal covariance matrices Σ_i and Σ_0.The stated diagonals depend on σ²_i(w_i, c_i, y_i) and σ²_0(α), respectively.Source: B Model details · p. 11

C Practicalities and Implementation

• VAE inference can suffer component collapse, producing redundant topics with many topics containing the same words.The implementation borrows a mitigation approach from Srivastava and Sutton (2017).Source: C Practicalities and Implementation · p. 11
• The implementation combines high-momentum Adam with batch normalization after µ, log σ², and η to avoid divergence.These are the specified locations for batch-normalization layers.Source: C Practicalities and Implementation · p. 11
• Batch normalization on η solves mode collapse but can encode word-distribution information outside topic weights and the background distribution, leaving θ not necessarily useful for topical content.This is described as a test-time issue caused by shifting inputs using the training population mean.Source: C Practicalities and Implementation · p. 11
• Annealing η from its batch-normalized output to its unnormalized output is intended to ensure interpretable final representations and avoid ProdLDA's possible background-term ambiguity.The procedure uses a convex combination of the two outputs.Source: C Practicalities and Implementation · p. 11

• Documents were lowercased and tokenized; punctuation, numeric tokens, tokens shorter than three characters, and Snowball stopwords were removed.The authors provide preprocessing scripts because these choices can substantially affect perplexity.Source: D Data · p. 11
• Vocabulary selection retained words occurring in the most documents, capped at 2,000 for 20 Newsgroups and 5,000 for the other datasets.Source: D Data · p. 11
• The Yahoo Answers file differed from its website description, so the authors dropped two undersized categories and merged two apparently equivalent arts categories, yielding 15 categories of 10,000 documents each.The documents were downloaded from the project webpage.Source: D Data · pp. 11–12

• All experiments used 300-dimensional word embeddings, with a softplus-based text encoder and a label decoder that outputs class scores.Training used one noise sample; test ELBO estimation used 20.Source: E Experimental Details · p. 12
• Unsupervised models used Adam with β1 = 0.99, learning rate 0.002, batch size 200, and 200 epochs, except Yahoo Answers used 150 epochs.LDA hyperparameters were updated every 10 epochs.Source: E Experimental Details · p. 12
• External NPMI used New York Times English Gigaword co-occurrences from 2000–2009, while classification compared logistic regression and the model on the same input data and vocabulary.Logistic-regression l2 strength was tuned by cross-validation; model epochs were selected on development data, without regularization or word vectors.Source: E Experimental Details · p. 12

• On 20 Newsgroups, SAGE and regularized SCHOLAR had better coherence than LDA, while NVDM still had worse perplexity, though less dramatically than previously reported.Word vectors did not improve internal coherence, perhaps because the dataset contains more names and specialized terminology; Table 4 reports performance for 20 topics and a 2,000-word vocabulary.Source: F Additional Experimental Results · p. 12, Table 4 · p. 12
• On Yahoo Answers, SCHOLAR had higher perplexity but higher internal coherence than LDA; word vectors improved internal and external coherence at the cost of worse perplexity.Without a background term, SCHOLAR achieved the best external coherence, but its topics gave high weight to common words and were more repetitive; Table 5 uses 250 topics and a 5,000-word vocabulary.Source: F Additional Experimental Results · p. 12, Table 5 · p. 12
• Table 6 presents unsupervised SCHOLAR topics for 20 Newsgroups alongside their internal coherence values.Source: Table 6 · p. 13