Source-linked AI summary
What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
Amritanshu Agrawal, Wei Fu, Tim Menzies
TL;DR
LDA can produce different topics when training-data order changes, threatening the reliability of analyses based on those topics. The paper introduces LDADE, which tunes LDA with differential evolution, and evaluates it across multiple software-engineering datasets and implementations. Across treatments, LDADE reduces topic instability and improves downstream classification performance, supporting automated tuning and stability checks for SE studies using LDA.
Problem
LDA's order effects can generate different topics from different input orderings, causing misleading topic descriptions and reducing classifier efficacy.
Method
LDADE combines LDA with differential evolution to automatically tune LDA parameters for each dataset.
Results
Across implementations, platforms, LDA variants, and datasets, LDADE dramatically reduces topic instability and improves classification performance.
Takeaways & Limitations
SE papers using LDA should test and, if needed, mitigate topic instability before relying on learned topics for conclusions.
Takeaways & Limitations
The evaluation uses topic similarity and F1/F2 measures, while other measures such as perplexity, performance, and accuracy could yield further evidence.
Abstract
from arXiv · showhide
Context: Topic modeling finds human-readable structures in unstructured textual data. A widely used topic modeler is Latent Dirichlet allocation. When run on different datasets, LDA suffers from "order effects" i.e. different topics are generated if the order of training data is shuffled. Such order effects introduce a systematic error for any study. This error can relate to misleading results;specifically, inaccurate topic descriptions and a reduction in the efficacy of text mining classification results. Objective: To provide a method in which distributions generated by LDA are more stable and can be used for further analysis. Method: We use LDADE, a search-based software engineering tool that tunes LDA's parameters using DE (Differential Evolution). LDADE is evaluated on data from a programmer information exchange site (Stackoverflow), title and abstract text of thousands ofSoftware Engineering (SE) papers, and software defect reports from NASA. Results were collected across different implementations of LDA (Python+Scikit-Learn, Scala+Spark); across different platforms (Linux, Macintosh) and for different kinds of LDAs (VEM,or using Gibbs sampling). Results were scored via topic stability and text mining classification accuracy. Results: In all treatments: (i) standard LDA exhibits very large topic instability; (ii) LDADE's tunings dramatically reduce cluster instability; (iii) LDADE also leads to improved performances for supervised as well as unsupervised learning. Conclusion: Due to topic instability, using standard LDA with its "off-the-shelf" settings should now be depreciated. Also, in future, we should require SE papers that use LDA to test and (if needed) mitigate LDA topic instability. Finally, LDADE is a candidate technology for effectively and efficiently reducing that instability.
1. Introduction
LDA is widely used to analyze unstructured software-engineering text, but stochastic training creates order effects that destabilize topics and can undermine downstream classification. The paper proposes LDADE, which automatically tunes LDA parameters using differential evolution, and evaluates it across datasets and implementations.
- Motivation: Unstructured software-engineering text is difficult to search, characterize, and classify, motivating topic modeling with LDA.The paper applies LDA to Stack Overflow, SE-paper abstracts and titles, and NASA defect reports.
- Problem: Different input orderings can produce different LDA topics because the algorithm uses stochastic, incremental updates.These order effects create systematic instability in topic-modeling analyses.
- Approach: LDADE combines LDA with differential evolution to automatically tune the parameters <k, α, β>.The authors prefer this search-based approach because it is faster than other methods and addresses order effects.
- Findings: The study reports that LDADE improves topic stability and classification performance compared with standard LDA.The evaluation uses F1 and F2 classification metrics, with F2 emphasizing avoidance of false positives.
- Implications: The authors conclude that SE studies using LDA should test and, when necessary, mitigate topic instability.They specifically recommend tuning before using learned topics to support conclusions.
- Problem: 25 of 40 reproduced Stack Overflow topics were unstable, with word overlap of 55% or less across two runs.The comparison changed only the randomly generated input ordering while keeping other parameter settings intact.
3. Related Work
The paper reviews how LDA represents documents and topics, why its stochastic updates can create order effects, and how tuning may stabilize results. It contrasts the proposed automated approach with predominantly manual, limited, or insufficiently stability-aware practices in prior work.
- 3.1. Topic Modeling: LDA explains observations through unobserved groups and learns topic, word-probability, word-topic, and document-topic distributions.Its scalability supports application to large software-engineering text corpora.
- 3.1. Topic Modeling: LDA uses α = P(k|d) for topic prevalence in documents and β = P(w|k) for word probabilities within topics.These distributions are initialized and iteratively updated through sampling.
- 3.1. Topic Modeling: Key LDA settings include k, the number of topics, burn-in iterations, and the sampling interval, alongside α and β.Earlier tuning work studied these settings largely through manual, expertise-guided processes.
- 3.2. Order Effects and Tuning: Random initialization and incremental resampling make LDA susceptible to order effects, while useful initializations may reduce them.The required initialization is dataset-specific, so tuning must be repeated for each new dataset.
- 3.3. Prior Work: Prior SE studies commonly mentioned instability but relied on off-the-shelf parameters, manual adjustment, or limited exploration.Among 57 reviewed SE-related papers, 28 mentioned instability and 10 of those still used off-the-shelf settings.
- 3.3. Prior Work: Manual topic inspection and user knowledge can address result stability, but these approaches require substantial effort or introduce subjectivity.The paper therefore favors fully automated methods for reproducibility.
- 3.3. Prior Work: Earlier genetic-algorithm approaches did not apply subsampling stability tests and could become much slower if stability objectives were added naively.LDADE is presented as preferable partly because it terminates quickly.
4. Methods
The study evaluates LDA stability across three unrelated software-engineering datasets and measures topic overlap across shuffled runs. LDADE combines LDA with differential evolution to tune parameters for more stable topics.
- Data Sets: Three unrelated datasets cover NASA defect reports, Stack Overflow programming discussions, and titles and abstracts from 15,121 software-engineering papers.The datasets support evaluation across distinct software-engineering tasks and data scales.
- Data Preparation: All datasets undergo stop-word removal, Porter stemming, and tf-idf feature selection before modeling.Tf-idf focuses on words frequent within relatively few documents.
- Similarity Scoring: Topic stability ℜ_n is measured as the median cross-run overlap of n topic words after repeatedly shuffling training-data order.The study evaluates n ≤ 9 and repeats stability calculations across multiple runs to reduce sampling bias.
- Similarity Scoring: The analysis distinguishes raw stability scores before and after tuning from Delta, defined as after-tuning minus before-tuning stability.The input ordering is shuffled for every LDA run to induce variance across orderings.
- Tuning Topic Modeling with LDADE: LDADE uses differential evolution to adjust LDA’s < k, α, β > parameters while maximizing similarity scores.The paper selects differential evolution after comparing search-based software-engineering optimizers and reports it as faster than alternatives.
- Implementation Variants: The evaluation includes deterministic VEM and stochastic Gibbs-sampling LDA implementations across different computing environments.Smaller datasets use Scikit-Learn and Python, while Stack Overflow uses Spark and Mllib on a 45-node cluster.
- Tuning Topic Modeling with LDADE: Differential evolution had not previously been applied in this work to tune LDA specifically for increased stability.Table 7 summarizes the LDADE algorithm.
5. Results
The results test whether input ordering changes LDA topics and compare untuned and tuned stability. Untuned LDA stability collapses beyond five topic words, undermining conventional longer topic descriptions.
- Order Effects: Changing the order of training data is tested as a source of substantial variation in topics learned by LDA.The analysis compares runs with different input orderings while keeping other parameter settings fixed.
- Untuned Stability: Untuned LDA stability collapses most after n = 5 words per topic.Reports using more than five words per topic can change solely because the input order changes.
- Untuned Stability: The instability conflicts with standard advice to report the top 10 words per LDA topic.The authors state that such 10-word topics would rarely recur across multiple runs.
Result 1
LDADE tuning substantially improves topic stability across datasets and overlap sizes. The strongest reported gains occur where standard LDA is especially unstable, including highly skewed data.
- Stability Improvement: LDADE tuning never reduces stability and often produces large positive gains, especially beyond five overlapping terms.In PitsD, improvement reaches 100% for overlaps up to eight terms.
- Stability Improvement: After tuning, topics of up to seven words recur across different input orderings in 66% of cases.This summarizes the majority-case stability reported across the evaluation.
- Dataset Effects: Highly skewed PitsD data achieved the largest improvement, although approximately 92% of that sample has severity level 3.The authors contrast this with mixed-severity Pits datasets, where LDADE achieves its highest improvement under less skewed data.
- Stable Topics: LDADE finds 27 topics that were unstable when produced by standard LDA.The corresponding table reports the percentage of topic words appearing in nearest matches across runs.
- Stable Topics: LDADE’s usual topic stability is nearly 88% or higher, with a worst observed stability of 55%.The worst tuned stability equals the median instability reported for standard LDA in Table 3.
Result 2
The classification experiments compare tuned and untuned topic-distribution features using F1 and F2. Tuning yields statistically significant improvements, with k identified as the most influential parameter for classification accuracy.
- Classification Setup: The experiments train a Linear Kernel SVM on document topic distributions after selecting tuned LDA configurations.The tuning objective remains maximizing ℜ_n, and the optimized k, α, and β are then used for classification.
- Metrics: F1 measures are plotted by dataset, while F2 measures weight recall more heavily than precision.Untuned 10 uses default k = 10; tuned k and tuned 10 use LDADE-tuned parameters under their respective k settings.
- Classification Results: Tuned and untuned classification results differ significantly at 99% confidence, with A12 ≥ 0.6 indicating more than a small effect.Scott-Knot ranking, A12 effect size, and bootstrapping agreed on the distinction.
Result 3
Tuning is highly recommended for SE classification tasks because the number of topics strongly affects classification accuracy. Across datasets, tuned parameter values vary substantially and can differ from standard recommendations.
- Tuning is highly recommended for SE classification tasks, with k having the greatest effect on classification accuracy.
- For n = 5 word overlap, Figures 10–12 report median results and interquartile ranges across 10 tunings.For PitsB, IQR = 0, indicating that tuning always converged on the same final value.
- Tuned parameter ranges vary by dataset, with IQR ranges often spanning about 50% of the median.
- Best results in these datasets used k ≤24 topics, contrasting with a recommendation of k = 67 for other data.
Result 4
LDADE’s stability improvements persist across datasets, implementations, platforms, and inference methods. The results support re-tuning LDA for every new dataset rather than reusing settings from prior studies.
- Tunings from other datasets should not be reused; every new dataset should be re-tuned.
- The evaluation compares Python/Scikit-Learn desktop runs with Scala/Mllib Spark-cluster runs across multiple datasets.
- Comparisons also vary LDA inference between VEM and Gibbs sampling.VEM is deterministic variational EM, whereas Gibbs sampling is an approximate stochastic process for computing and updating α and β.
- Across implementations and inference methods, tuning never makes stability worse and sometimes dramatically improves it.
- Tuning appears more important for VEM than Gibbs, with the largest post-tuning improvements reported for VEM.The paper attributes this difference to VEM reaching a local optimum dependent on initialization and other optimization factors.
Result 5
Instability is consistent across LDA implementations rather than being an implementation-specific quirk, while LDADE stabilizes the resulting models. Differential Evolution settings are examined as part of the tuning process.
- Instability is not due to a quirk of LDA implementation; it is consistent, and LDADE can stabilize LDA.
- Differential Evolution’s recommended population size is np = 30 when tuning k, α, and β, compared with np = 10 used in the paper’s algorithm.
Result 6
Stable topic-model parameters are easier to find than parameters in many standard optimization tasks, but search-based tuning adds runtime cost. In these experiments, LDADE increased runtime by up to a factor of five.
- Finding stable topic-model parameters is easier than solving standard optimization tasks.
- Search-based SE methods can be very slow, with prior work requiring 15 years of CPU time or 10^6 evaluations.
- The experiments predict tuning will take three times longer because Differential Evolution runs for three generations.
- Tuning slows LDA by up to a factor of five, close to the theoretical prediction.LDADE runtimes include the time required for Differential Evolution to find the tunings.
Result 7
LDADE is compared with LDA-GA and random search on stability, runtime, and classification performance. Across these comparisons, LDADE generally produces stabler topics and faster or slightly better results, while tuning costs more than untuned LDA.
- LDADE costs three to five times more runtime than untuned LDA.
- 7/10 datasets show somewhat bigger classification performance for LDADE than LDA-GA, although overall F1 performances are similar.
- LDADE's median classification performance is slightly better than LDA-GA's.
- 24-70 times faster runtime is reported for LDADE than LDA-GA.
- LDADE produces stabler topics than both LDA-GA and random search.
- Best LDA tunings are dataset-specific, and off-the-shelf settings may fall far from the useful range.
Result 9
The paper recommends tuning LDA before using its topics or topic distributions because untuned topics are unstable. Its conclusions remain bounded by dataset, learner, evaluation-measure, and task coverage limitations.
- Off-the-shelf LDA is not recommended for learned topics or cluster distributions used in classification.
- Threats to Validity: The findings may not generalize beyond the finite datasets used, whose preprocessing and composition could affect results.
- Threats to Validity: Results used default secondary LDA parameters, a linear-kernel SVM, and only a small subset of available data miners.
- Threats to Validity: Evaluation relied on topic similarity and F2, while other software-engineering measures may produce different assessments.
- LDA studies should tune topics before drawing conclusions or using topic distributions in further tasks.
- Up to eight topic words can be reliably reported, but only after tuning for stability.