Variational Pretraining for Semi-supervised Text Classification

• VAMPIRE combines variational document modeling with language-model pretraining insights, using bag-of-words input to reduce sequence-model costs while learning downstream representations.The paper also reports that topic coherence is a better model-selection criterion for downstream classification than validation-set perplexity.Source: §1 · p. 1
• The evaluation compares VAMPIRE on four text-classification datasets against self-training, in-domain representation-learning methods, and contextual embeddings, including fine-tuned models.The experiments vary the amount of labeled data.Source: §1 · p. 1
• When domain-fine-tuned BERT is unavailable, VAMPIRE matches or exceeds alternatives with minimal computation, especially with limited labeled data.The authors report that fine-tuned BERT provides the best results when available.Source: §1 · pp. 1–2
• The paper adapts variational document models to modern pretraining for semi-supervised classification and highlights model-selection criteria as a contribution.Source: §1 · p. 2
• The authors identify fine-tuning contextual embeddings as essential for document classification and offer practical advice for using unlabeled data.Source: §1 · p. 2
• The authors release code for pretraining variational models on unlabeled data and using their learned representations in downstream tasks.Source: §1 · p. 2

2.1 Resource-limited Environments

• Low-resource settings involve constraints on computation, labels, and out-of-domain data; data and computational availability also vary across languages and domains.Limited labels may require expensive expert annotation, while incorporating contextual embeddings can be prohibitively expensive for researchers with scarce computation.Source: §2.1 · p. 2
• The paper seeks an approach that uses in-domain unlabeled data, works with few labels, and runs efficiently on a CPU.Source: §2.1 · p. 2

2.2 Semi-supervised Learning

• Contextual embeddings can require hundreds of millions of words and extensive GPU training, while sequence-to-sequence VAEs raise practical concerns about training difficulty and cost.The cited sequence-to-sequence VAE studies include Xu et al. (2017) and Yang et al. (2017); the training-difficulty concern cites Bowman et al. (2016).Source: §2.2 · p. 2
• The paper instead explores neural document models for semi-supervised learning, emphasizing faster training and an explicit topic interpretation.The cited models include Miao et al. (2016), Srivastava and Sutton (2017), and Card et al. (2018).Source: §2.2 · p. 2

3 Model

• Documents have labeled and unlabeled examples from the same distribution, and the goal is to learn a probabilistic classifier p(y | x).The model pretrains on unlabeled documents before using learned representations in supervised learning.Source: §3 · pp. 2–3, §3.1 · p. 3

3.1 Unsupervised Pretraining

• The VAE maps word counts to latent representations and reconstructs word-frequency inputs with a softmax decoder, avoiding sequence-based inference and enabling a lightweight unigram model.Its latent topic distribution is interpreted through a softmax.Source: Figure 1 · p. 3, §3.1 · p. 4

3.2 Model Selection via Topic Coherence

• NPMI topic coherence or log likelihood can guide early stopping; NPMI is averaged across topics and training stops when it ceases increasing for a preset number of epochs.Topic coherence is considered because optimizing an unlabeled objective to convergence may not yield the best downstream representations.Source: §3.2 · p. 4

3.3 Using a Pretrained VAE for Text Classification

• The downstream classifier learns a weighted combination of the VAE’s topic representation and encoder states, concatenated with sequence-to-vector representations.The combination uses softmax-normalized trainable weights; joint training and combined approaches did not reliably improve on pretraining.Source: §3.3 · p. 4

3.4 Optimization

• Training uses Adam, reconstruction batch normalization, and KL annealing; the feedforward architecture can be parallelized and run efficiently on CPUs or GPUs.The KL weight increases gradually from zero to one.Source: §3.4 · p. 5

4 Experimental Setup

• Experiments vary labeled training data from 200 to 10,000 instances and assume 75,000 to 125,000 unlabeled in-domain examples, reporting mean test performance across five random seeds.Each seed uses a different labeled sample.Source: §4 · p. 5

4.1 Datasets and Preprocessing

• The four classification datasets are AG News, IMDB, YAHOO! Answers, and tweets labeled across four HATESPEECH categories.Evaluation uses official test sets or stratified 25,000-document test samples, plus 5,000 validation instances.Source: §4.1 · p. 5

4.2 VAMPIRE Architecture

• For each dataset, random search selects the VAMPIRE pretraining model with the best NPMI for downstream classification.Hyperparameter sampling bounds and final assignments are reported in Appendix A.1.Source: §4.2 · p. 5

4.3 Downstream Classifiers

• The baseline downstream encoder is a Deep Averaging Network that sums token embeddings and passes them through a multilayer perceptron.Preliminary experiments found one-layer MLPs with moderate dropout more reliable on validation data than CNNs or LSTMs, especially with few labels.Source: §4.3 · p. 5

4.4 Resources and Baselines

• Low-resource comparisons include supervised and GloVe models and self-training; high-resource comparisons include frozen or fine-tuned ELMo and BERT, plus in-domain-only ELMo.The low-resource setting assumes limited computational resources, while high-resource models use plentiful compute and large out-of-domain data.Source: §4.4 · pp. 5–6

5 Results

• VAMPIRE achieves the highest accuracy among the considered low-resource methods, especially when labeled data are scarce.Table 2 reports test accuracies across four datasets and training sets of 200, 500, 2,500, and 10,000 documents.Source: §5 · p. 6, Table 2 · p. 7
• Fine-tuning pretrained BERT to in-domain data provides the best performance in the high-resource setting.Frozen off-the-shelf BERT and ELMO perform poorly relative to domain adaptation, especially on HATESPEECH and IMDB.Source: §5 · pp. 6–7
• For HATESPEECH and IMDB, ELMO trained on in-domain data performs far better than frozen off-the-shelf ELMO.The difference between these approaches is smaller for YAHOO! and AG.Source: §5 · p. 7
• The results indicate that unlabeled in-domain data is important for fine-tuning pretrained language models in high-resource settings and training VAMPIRE in low-resource settings.The paper contrasts these uses with the gains from pretraining on large amounts of web text over purely supervised models.Source: §5 · p. 7

6.1 NPMI versus NLL as Stopping Criteria

• Better pretrained VAMPIRE models tend to yield better downstream performance, while NPMI and NLL values are negatively correlated (ρ = –0.72).Downstream performance correlations are ρ = 0.55 for NPMI and ρ = –0.53 for NLL.Source: §6.1 · pp. 7–8
• NPMI-selected VAMPIRE models have lower variance in downstream accuracy than NLL-selected models, making NPMI the preferred selection criterion.The comparison uses 200 labeled IMDB documents; all VAMPIRE results in Table 2 use NPMI-selected models.Source: §6.1 · p. 8, Figure 4 · p. 8

6.2 Learned Latent Topics

• VAMPIRE’s document representations can be interpreted in terms of topics, which help summarize what pretraining learned.The paper provides example topics from IMDB and YAHOO!.Source: §6.2 · p. 8, Table 3 · p. 8

6.3 Learned Scalar Layer Weights

• The classifier consistently upweights the VAE’s first encoder layer and θ, and initializing their scalar weights highly improves validation performance, especially in low-resource settings.The intermediate encoder layers are downweighted, although using a multi-layer encoder yields larger downstream gains.Source: §6.3 · p. 8

6.4 Computational Requirements

• VAMPIRE can be trained orders of magnitude faster than ELMO, including without expensive hardware.On a GPU, VAMPIRE uses about 750MB of memory compared with ELMO’s 8.5GB for the reported IMDB pretraining setup.Source: §6.4 · p. 9, Table 4 · p. 9

7 Related Work

• Prior limited-label work found deep-model pretraining and fine-tuning important for semi-supervised speech recognition, and reported large gains from adversarial pretraining.Dataless classification uses category names as the only supervision.Source: §7 · p. 9
• Active learning seeks to maximize performance when labels are costly, while data augmentation work includes back translation and tf-idf word replacement.Source: §7 · p. 9
• The paper states that these prior approaches could be productively combined with its proposed methods.Source: §7 · p. 9

8 Recommendations

• With unlimited resources, pretrained models such as BERT currently give the best results, and fine-tuning to in-domain data is critically important.Source: §8 · p. 9
• When computation and annotations are limited but unlabeled data is plentiful, VAMPIRE offers large gains over other low-resource approaches.Source: §8 · p. 9
• Training ELMO only on in-domain data can match or somewhat outperform VAMPIRE, but may be prohibitively expensive without GPUs.Source: §8 · p. 9
• With tens of thousands of labeled instances, the advantages of additional unlabeled data become negligible, though other NLP tasks may have different tradeoffs.Source: §8 · p. 9

9 Conclusions

• With limited labels, VAMPIRE beats self-training and out-of-domain word vectors and approaches in-domain ELMO at a fraction of the computational cost.The authors position it as an alternative when BERT cannot readily be used because of computational limits or the absence of an appropriate pretrained model in the relevant language.Source: §9 · p. 9

A Hyperparameter Search

• The appendix describes a hyperparameter search used to choose model configurations.The search section includes plots of validation-performance ranges across settings.Source: A Hyperparameter Search · p. 12

A.1 VAMPIRE Search

• For VAMPIRE, the search varied hyperparameters across several dimensions, with the assignments reported in Table 5.Source: §A.1 · p. 12

A.2 Classifier Search

• Under low-resource conditions, logistic regression and DAN were more reliable, while LSTM validation accuracy had extremely high variance.With enough compute, CNNs could be marginally higher, but their accuracy distribution was mostly below those of logistic regression and DAN.Source: §A.2 · p. 12
• The authors selected DAN for pretrained-model experiments because it combines rich vector representations with reliability and few hyperparameters to tune.The search and manual tuning compared CNN, LSTM, Bag-of-Embeddings, and logistic regression classifiers.Source: §A.2 · p. 12

C Further Details on NPMI vs. NLL as Stopping Criteria

• Overtraining can lower NPMI while NLL continues to improve, so model selection should use NPMI to avoid poor topic coherence.Source: Figure 5 · p. 12, C Further Details on NPMI vs. NLL as… · p. 12

B Results in the High Resource Setting

• On AG News, frozen ELMO tended to outperform VAMPIRE; the authors suggest its news-crawl pretraining may explain its smaller fine-tuning gap.Source: B Results in the High Resource Setting · p. 12
• In-domain-only ELMO matched or beat off-the-shelf ELMO or BERT, while fine-tuned BERT led except on YAHOO! with 200 labeled instances.The authors attribute that exception to potentially improvable hyperparameter choices.Source: B Results in the High Resource Setting · p. 12, Table 6 · p. 14

D Additional Learned Topics

• The appendix presents additional examples of topics learned by VAMPIRE on the YAHOO! dataset.Source: D Additional Learned Topics · p. 12, Table 7 · p. 14