Topic Memory Networks for Short Text Classification

• Short texts provide limited classifier features because of data sparsity, while external-resource approaches depend on high-quality data that may be unavailable for some domains or languages.Source: §1 · p. 1
• The tweet example illustrates that word overlap with a differently labeled instance can mislead classification, while a word’s co-occurrence with “Bieber” can indicate the test text’s category.Source: §1 · pp. 1–2
• Topic memory networks use document-level word co-occurrence to identify class-indicative words through latent topics, without relying on external knowledge.Source: §1 · p. 2
• Unlike methods that use pre-trained topics as features, TMN encodes latent topic representations via memory networks to allow joint inference of topics.Source: §1 · p. 2
• On four benchmark datasets, the model outperforms state-of-the-art counterparts and generates meaningful, category-indicative topics.Source: §1 · p. 2

2 Topic Memory Networks

• TMN processes each short text as bag-of-words and sequence inputs through a topic model, topic memory, and configurable classifier, whose components update jointly.The classifier may use options such as CNN or RNN, depending on the application scenario.Source: §2 · p. 2

2.1 Neural Topic Model

• The neural topic model uses a K-dimensional latent variable z to form topic mixtures and word distributions, then applies variational inference with a negative variational lower-bound loss.The generation process draws z, transforms it into topic mixture θ, and generates words from topic-conditioned distributions.Source: §2.1 · pp. 2–3

2.2 Topic Memory Mechanism

• Topic memory maps NTM topics to classifier features by matching topics to embedded words, integrating matches with θ, weighting target memories into R, then joining R with sequence embeddings.The source and target memory matrices are produced from the topic-word weight matrix.Source: §2.2 · p. 3

2.3 Joint Learning

• The three modules update simultaneously under a combined objective that adds the NTM loss to classification cross-entropy, balanced by trade-off parameter λ.The framework jointly tackles topic modeling and classification.Source: §2.3 · p. 4

3.1 Datasets

• Four datasets comprise Snippets (8 labels), TagMyNews (7), Twitter (50 selected hashtags), and Chinese Weibo (50 hashtag labels); Weibo conversations were excluded.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 Model Settings

• Data were split 80/20 for training and testing, with 20% of training held out for development; the experiments used CNN with 500 hidden units, 200-dimensional embeddings, and K=50 topics.Topic-model BoW inputs excluded stop words and words occurring fewer than three times, whereas classification inputs used raw word sequences.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 Comparison Models

• Both TMN variants—using separate or joint topic inference—significantly outperform all comparison models (p < 0.05, paired t-test).Source: Table 3 · p. 5
• Comparisons span SVM and topic-model baselines, AttBiLSTM and CNN, CNN+TEWE, CNN+NTM, and TMN with separate versus joint topic inference.Source: §3.3 · p. 5

4.1 Classification Comparison

• TMN significantly outperforms the best comparison models on all four datasets, including models that concatenate topic features.The comparison includes TMN variants that infer topics separately or jointly with classification.Source: §4.1 · p. 6

4.2 Topic Coherence Comparison

• TMN achieves higher topic-coherence CV scores than the compared models, while its sample Egyptian-revolution topic is described as the best.The sample topic includes indicative terms such as “tahrir” and “cairo”; NTM also scores better than LDA and BTM.Source: Table 4 · p. 6, Table 5 · p. 6, §4.2 · p. 7

4.3 Results with Varying Hyperparameters

• Accuracy gains from additional hops generally saturate at five, while TMN remains more accurate than CNN+TEWE across tested topic counts.Five hops performs best on most datasets except Snippets; topic-count curves are nonmonotonic, with K=50 best for TMN on TagMyNews.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4 A Case Study on Topic Memory

• For test instance S, topic memory links “wristbands” to label-indicative latent topics through corpus co-occurrence, despite S lacking their top words.It also distinguishes topical words such as “wristband” from background words such as “i” and “for”.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

4.5 Error Analysis

• TagMyNews errors include confusing context-dependent word senses and failing to capture phrase-level meanings.The authors suggest context-sensitive topical word embeddings and phrase-discovery topic models as future directions.Source: §4.5 · p. 8

Short Text Classification

• Short-text classification research has used external knowledge, manually crafted task-specific features, and deep-learning representation learning.The cited approaches vary in resource reliance and whether they require feature engineering.Source: §5 · p. 8
• Unlike approaches using pre-trained topic mixtures as features, TMN encodes topics in memory and jointly induces them with classification end to end.The paper states that its framework learns effective representations from internal data.Source: §5 · p. 8

Topic Models

• Topic-model research spans pLSA and LDA, short-text variants such as BTM and LeadLDA, and neural topic models, including VAE-style models.The paper focuses on VAE-style neural topic modeling for short-text classification.Source: §5 · p. 9

• The model achieves better results than state-of-the-art models on short text classification and topic coherence evaluation across four benchmark datasets.Source: §6 · p. 9
• Topic memory networks exploit corpus-level topic representations through a topic memory mechanism for short text classification.Source: §6 · p. 9
• The model alleviates data sparsity by jointly learning latent topics and text categories.Source: §6 · p. 9