PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Top Words: MonoTM for Topic Modeling with Interpretable Monosemantic Features

Una Joh, Bei Yu

arXiv:2609.09575v1cs.CLcs.HC

TL;DR

Topic models often describe topics with words, while analysts need more meaningful semantic units and clearer evidence about how interpretability relates to topic inference. MonoTM uses sparse autoencoders over document embeddings, separating full-feature mixture estimation from validated semantic topic descriptors. Across three benchmark corpora, the paper finds that these two roles favor different SAE configurations and feature subsets, motivating the framework’s decoupled design.

  • Problem

    Topic models typically describe topics with top-ranked words, but words can be inadequate semantic units for meaningful topic interpretation and downstream corpus analysis.

  • Method

    MonoTM estimates document–topic mixtures from the full SAE bag-of-features representation and learns topic descriptors over separately validated semantic features with mixtures fixed.

  • Results

    Across three benchmark corpora, document–topic mixture estimation and semantic interpretation favor different SAE configurations and feature subsets.

  • Takeaways & Limitations

    Decoupling statistical and interpretive roles preserves global topic structure while enabling topic descriptions with corpus-grounded semantic units beyond individual words.

  • Takeaways & Limitations

    MonoTM’s validated feature set depends on LLM capabilities, prompts, and decoding settings, so included features and apparent granularity may change across models.

Abstract

from arXiv · show

Topic models summarize large text corpora, but top-ranked words often provide only a limited representation of topic semantics. Sparse autoencoders (SAEs) offer a way to move beyond word-level descriptors by extracting interpretable features from dense representations, yet how feature interpretability relates to topic-inference quality remains unclear. We introduce \textbf{MonoTM}, an interpretable topic modeling framework that decouples these roles. Across three benchmark corpora, we show that document--topic mixture estimation and semantic interpretation favor different SAE configurations and feature subsets. MonoTM estimates mixtures from the full SAE bag-of-features representation and, with them fixed, learns topic descriptors over a separate vocabulary of corpus-grounded semantic features. This design preserves global topic structure while representing topics with semantic units more meaningful than individual words, making them more useful for downstream corpus analysis.

1 Introduction

Topic modeling supports large-scale corpus exploration, but word-based topic descriptors often fall short of the semantic units analysts need. MonoTM addresses this gap by extracting human-readable features from document embeddings while retaining competitive document–topic mixture estimation.

  • Motivation: Topic modeling is widely used for exploratory corpus mapping and thematic summarization, but its role as scholarly evidence remains contested.Validation and reporting practices remain heterogeneous across computational social science studies.
  • Motivation: Most topic models define topics as word distributions, making concise, human-readable topic labels difficult to produce.Neural topic models add semantic information but typically still describe topics with top-ranked words or phrases.
  • Motivation: Words are often inadequate semantic units for interpreting topics or analyzing relations between topics and documents.Analysts may need an intermediate representation between individual documents and coarse corpus-level themes.
  • Contribution: MonoTM extracts human-readable semantic features from document embeddings and achieves competitive document–topic mixture estimation.The framework is motivated by sparse autoencoders and autointerpretability methods.

2 Background & Related Work

Mechanistic interpretability and sparse autoencoders provide a foundation for representing documents with reusable semantic features rather than only words. MonoTM builds on this work by separating SAE features’ statistical role in topic modeling from their interpretive role.

  • Mechanistic Interpretability: Mechanistic interpretability explains neural networks through internal components and motivates learned feature directions because neurons can be polysemantic.Superposition explains how sparse factors can encode more features than the model has dimensions.
  • Mechanistic Interpretability: Sparse autoencoders learn sparse, overcomplete feature bases that can produce more monosemantic units than neurons and support meaningful interventions.Language models can also generate and score natural-language explanations of internal directions.
  • MonoTM’s Position: This paper applies SAEs to document embeddings as reusable semantic units and uses an Interpreter–Predictor protocol to validate feature labels.The protocol is intended to support topic description with corpus-grounded semantic features.
  • Topic Modeling: Earlier topic models represent documents as topic mixtures and topics as word distributions, while neural variants add inference networks, embeddings, or contextual semantics.These approaches retain interpretability but commonly describe topics through words or phrases.
  • MonoTM’s Position: Unlike approaches that simply replace words with SAE latents, MonoTM decouples features useful for mixture estimation from features whose labels are reliably interpretable.The design prioritizes corpus-specific interpretability and validation over fast topic discovery in some applications.

3 Research Questions

The paper hypothesizes that SAE features can serve both as semantic units and as statistical units for topic modeling, while recognizing that these roles may not align. It therefore asks how SAE configurations affect interpretability, mixture estimation, and integrated topic modeling.

  • Hypothesis: SAE features may provide human-readable semantic units and useful document–topic mixture estimates when document embeddings decompose into sparse, nonnegative feature combinations.The paper treats these roles as related but not identical.
  • Research Questions: The first research question asks which SAE configurations produce reliably interpretable semantic units.This targets the conditions under which corpus-trained features become useful for interpretation.
  • Research Questions: The second research question asks which SAE configurations make features effective units for estimating document–topic mixtures.This isolates the statistical role of SAE features from their interpretive role.
  • Research Questions: The third research question asks how SAE features can yield interpretable topic descriptors aligned with strong document–topic mixture estimates.This question motivates a framework that addresses both roles together.

4 Experimental Setup

MonoTM is evaluated on three benchmark corpora using sparse autoencoders over document embeddings and an Interpreter–Predictor protocol for feature-label validation. The setup varies SAE dictionary capacity and activity, then scores labels by document-level activation prediction.

  • Datasets: The evaluation uses 20 Newsgroups, Web of Science, and Reuters, comparing inferred topics with benchmark category annotations.Reuters is multi-label and imbalanced; retaining labels appearing at least 30 times yields 47 topics.
  • Document Representations: Each document is mapped to a 4096-dimensional embedding, with documents longer than 32,768 tokens truncated to the model’s context window.Only 18 20 Newsgroups documents are affected by truncation.
  • SAE Training: Each top-K SAE learns mN latent features, produces nonnegative activations, retains the largest K per document, and reconstructs the standardized embedding.N controls dictionary size, while K controls the maximum number of active features per document.
  • Feature Interpretation: The Interpreter generates short feature labels from maximal, typical, and zero-activation document examples.Labels aim to capture a prominent concept present in activating texts and absent from non-activating texts.
  • Feature Validation: The Predictor validates each label through independent per-document binary classification and computes feature interpretability as an F1 score.It receives only the candidate description and raw document text for each evaluation example.
  • Evaluation Protocol: Autointerpretability is applied only to selected SAE configurations because the Interpreter–Predictor protocol requires many LLM inference calls.The sweep includes N ∈ {0.5, 1, 2, 3, 4, 5} and K ∈ {4, 8, 16, 32}.

5 Results

Across three benchmarks, SAE feature configurations support different goals: interpretability favors low activity and smaller dictionaries, while mixture estimation favors balanced scaling and can improve as validated-feature fractions decline. MonoTM therefore separates mixture estimation from topic interpretation, using full bag-of-features representations for mixtures and a separate validated feature vocabulary for descriptors.

  • RQ1: Interpretability: Across all three datasets, smaller K yields the highest fraction of activated SAE features that pass interpretability validation.Larger K increases the absolute number of validated features at permissive thresholds, but the validated fraction decreases monotonically with K.
  • RQ1: Interpretability: The effect of dictionary capacity N depends on activity K and corpus: larger dictionaries can reduce validated counts at low K but expose more validated units at higher K.Web of Science is an exception at low K, where activated and validated counts increase with N.
  • RQ1: Interpretability: For practical feature vocabularies, low or moderate activity such as K = 4 or K = 8 favors fewer, more readily validated features, while larger N and K broaden coverage.The paper recommends starting with a high-precision configuration and expanding only when the validated inventory is too coarse or small.
  • RQ3: MonoTM: Mixture quality and feature interpretability favor different SAE configurations: in 20 Newsgroups, increasing K from 8 to 32 raises Micro/Macro F1 from 0.56/0.52 to 0.66/0.63 while validated-feature fraction falls from 35.2% to 13.8%.Thus, neither the absolute number nor dictionary-level fraction of validated features fully predicts document–topic mixture quality.
  • RQ2: Mixture estimation: BoF+LDA is competitive across all three benchmarks, with strongest results on both single-label datasets and the best Macro-F1 on Reuters.It replaces words with active SAE features weighted by activation magnitudes, treating them as pseudo-tokens for LDA.
  • RQ2: Mixture estimation: BoF+LDA achieves the best Macro-F1 overall despite BoW-LDA having the highest Micro-F1 on imbalanced Reuters, indicating stronger evidence for minority topics.Micro-F1 is dominated by frequent classes, whereas Macro-F1 is sensitive to rare labels.
  • RQ3: MonoTM: MonoTM decouples document–topic mixture estimation from interpretation by using the full SAE bag-of-features representation for mixtures and a separate validated feature vocabulary for descriptors.This design addresses the empirical mismatch between features useful for inference and features whose labels are reliably interpretable.
  • RQ3: MonoTM: The audit finds nearest validated same-topic descriptors much closer to excluded high-association features than random same-topic descriptors, especially for 20 Newsgroups and Web of Science.In Reuters, 70.7% of excluded features are flagged as format, register, or numeric cues rather than missing core topic semantics.

6 Conclusion

MonoTM shows that SAE features support multiple, distinct roles in topic modeling: mixture estimation and semantic interpretation do not necessarily favor the same configurations. It resolves this mismatch by estimating mixtures from the full SAE representation and learning topic-feature distributions over validated features with mixtures fixed.

  • SAE features support topic modeling in multiple but distinct ways, including semantic interpretation and document–topic mixture estimation.
  • The SAE configurations suited to feature interpretability and document–topic mixture estimation do not necessarily coincide.
  • MonoTM estimates document–topic mixtures from the full SAE representation, then learns topic–feature distributions over validated features with the mixtures fixed.
  • Excluded high-association features are typically covered by validated same-topic descriptors or reflect auxiliary non-core cues.

7 Limitations

MonoTM's interpretability has several scope and practicality constraints. Excluded features may contain poorly captured or culturally specific semantics, LLM-based validation can introduce blind spots and bias, and the labeling process is computationally expensive.

  • Excluded features are not proven unimportant or fully understood and may encode corpus-specific, pragmatic, formatting, or culturally specific concepts.The paper recommends coverage audits and, when appropriate, human expert review for high-impact applications.
  • Interpretability scores depend on the capabilities, failure modes, prompts, and decoding settings of the specific Interpreter and Predictor LLMs.Different LLM backends may vary in abstraction, binary-task calibration, and robustness to domain-specific language.
  • LLM-heavy feature labeling is computationally expensive and can be slow, creating a barrier for iterative modeling workflows.The paper identifies cheaper screening models and Predictor distillation as possible efficiency improvements.
  • Systematic LLM blind spots could disproportionately exclude features tied to particular registers, dialects, domains, or culturally specific concepts.Such exclusions could omit important topic aspects even when those aspects strongly influence inferred mixtures.
  • Table 3 provides dataset summary statistics, including data points, categories, label coverage, multi-label status, and document-length distributions.

B SAE Architecture and Training Details

MonoTM trains top-K sparse autoencoders over standardized document embeddings across a grid of dictionary expansions and activity levels, then evaluates feature-space reconstructability with directional linear probes.

  • SAE architecture: Each SAE uses a dictionary with mN latent features, tied encoder weights, nonnegative activations, and a hard top-K sparsity constraint.The embedding dimension is m = 4096, while N controls dictionary size and K limits active features per document.
  • Training procedure: Decoder columns are normalized after updates, and models are trained with Adam at learning rate 10^-4 and batch size 1024 for up to 300k steps.The selected checkpoint minimizes normalized reconstruction error on a fixed monitoring split containing 10% of documents.
  • Training grid: The experiment varies N and K over up to 11 × 8 = 88 configurations, omitting cases with mN < K while retaining mN = K as a dense diagnostic.The activity grid is K ∈ {4, 8, 16, 32, 64, 128, 256, 512}.
  • Cross-SAE probe: Validated feature activations are assembled into dense document–feature matrices, and affine ridge maps are fit in both directions between SAE configurations.The probe retains only features passing IS ≥0.80 and evaluates mappings with weighted MSE normalized by a zero-predictor baseline.
  • Cross-SAE probe: Directional asymmetry is defined as ∆ = ρ(big →small) −ρ(small →big), where negative values indicate easier reconstruction from the larger SAE to the smaller SAE.The procedure uses 10 random document train/validation splits and evaluates all ordered pairs among four configurations.

F.7 Results

Higher-capacity SAE representations generally reconstruct lower-capacity validated-feature spaces more easily, while mixture quality favors balanced sparsity rather than maximal interpretability or unconstrained capacity.

  • Cross-SAE predictability: On Web of Science and Reuters, higher-capacity configurations consistently achieve lower normalized cross-SAE reconstruction error than reverse mappings.Across configuration pairs and seeds, the median ∆ is negative on both datasets.
  • Cross-SAE predictability: On 20 Newsgroups, the aggregate median ∆ is also negative but less conclusive under resampling, with the strongest asymmetry between (N=2, K=4) and (N=5, K=32).The result is consistent with a larger representational-resolution gap between the most separated configurations.
  • Cross-SAE predictability: The cross-SAE probe supports a granularity shift in which larger SAEs contain enough information to linearly reconstruct validated-feature activations from smaller SAEs, but not conversely.The probe is a diagnostic of representational resolution rather than evidence of one-to-one feature splitting.
  • Mixture quality: Very wide SAEs with very small K and boundary cases where mN = K tend to produce weak document–topic mixture quality.The first regime yields too little per-document evidence, while the second removes the intended sparsity bottleneck.
  • Mixture quality: Stronger mixture performance appears when N and K increase together, preserving sparsity while allowing documents to express richer SAE-feature sets.The robust balanced region is used to summarize BoF+LDA performance in Table 1.

J Sample Topic–Feature Representations

MonoTM estimates document–topic mixtures from the full SAE bag-of-features representation, then learns topic–feature distributions over separately validated semantic features with mixtures fixed.

  • Sample representations: Even the relatively small (N=1, K=8) configuration yields topic feature labels that align with gold topics and provide finer-grained corpus descriptions.Larger settings such as (N=5, K=32) produce substantially more interpretable features, but are not required for the reported sample representations.
  • MonoTM stages: Mixture estimation applies LDA to the full sparse activation matrix rather than filtering to validated features.This stage uses all active SAE features because strict interpretability filtering generally hurts mixture estimation.
  • MonoTM stages: For topic description, MonoTM retains interpretation features that activate in at least 30 documents, have labels, and satisfy IS(f) ≥0.8.These features form the interpretable document–feature matrix C.
  • MonoTM stages: With Θ and C fixed, MonoTM estimates topic–feature distributions B and ranks each topic by the probabilities of its validated feature labels.The reported descriptors are the top-ranked labels under βt,f.
  • MonoTM design: MonoTM can use different SAEs for mixture estimation and interpretation, separating the representation optimized for topic mixtures from the one optimized for validated feature labels.The single-SAE case is recovered when the two representations are identical.

L Excluded-feature audit details

The excluded-feature audit tests whether lower-validation SAE features contain topic information missing from MonoTM’s validated descriptor vocabulary. It finds that many high-association exclusions overlap validated descriptors, while Reuters exclusions often reflect reporting format or financial-news register rather than substantive topics.

  • Audit construction: The audit constructs an all-active interpretation-feature matrix and estimates an auxiliary topic–feature distribution using fixed document–topic mixtures.The audit excludes interpretability-score weighting from activation measurement, while final MonoTM descriptors remain restricted to validated features.
  • Coverage analysis: Coverage is evaluated across candidate-pool settings M = 20, M = 50, and M = 100 using activation cosine to the nearest validated same-topic descriptor and a random validated baseline.These settings test increasingly broad validated descriptor regions.
  • Audit construction: High-risk excluded features are those with IS(f) < .8 and best all-feature topic rank at most 20.Candidate pools of validated same-topic descriptors are then used to test whether excluded features are covered.
  • Findings: In 20 Newsgroups and Web of Science, top-ranked excluded features closely match validated same-topic descriptors, with activation cosines of .650 and .732.The examples concern Israeli-Palestinian political criticism and rainwater harvesting, respectively.
  • Findings: Reuters exclusions are systematic and topic-associated but primarily encode reporting format or financial-news register rather than substantive topical content.Heuristic label-type flags are descriptive only, not manual annotations or validation metrics.

O Additional Descriptor Comparison Results

The controlled descriptor comparison holds document–topic mixtures fixed while comparing word-based, LLM-generated, and MonoTM feature-based topic descriptions. The resulting tables show distinct descriptor roles, with MonoTM using validated SAE features rather than lexical terms or free-form summaries.

  • Controlled comparison: All descriptor methods describe the same inferred topics under fixed document–topic mixtures Θ, enabling a controlled comparison across datasets.Reference labels are shown only for orientation, and Reuters labels are not exhaustive because the corpus is multi-label and imbalanced.
  • Descriptor roles: Word-based descriptors expose lexical anchors or named entities, whereas free-form LLM descriptors often summarize salient document clusters.The comparison covers θ-weighted TF-IDF and c-TF-IDF terms alongside free-form LLM outputs.
  • Descriptor roles: MonoTM descriptors are selected from a fixed vocabulary of validated SAE features and ranked by a topic–feature distribution.This provides a semantic-feature alternative to word-based and free-form descriptor mechanisms.

Q Affordance Demonstration: Topic Relations in a Shared Semantic-Feature Space

MonoTM uses a shared validated SAE-feature vocabulary to compare topics as well as describe them. In a Web of Science demonstration, feature-based and lexical representations produce different topic-relation structures and explanations.

  • Construction: The demonstration reuses fixed Web of Science document–topic mixtures and compares MonoTM’s validated feature space with a θ-weighted TF-IDF lexical space.MonoTM retains the top 200 validated features per topic, while the lexical analogue uses the top 50 terms.
  • Overlap analysis: Topic-pair overlap is computed from shared representation mass, with shared items ranked by the minimum topic-specific weight to explain each edge.The same overlap framework applies to validated SAE features and lexical terms.
  • Visualization: Figures 9 and 10 visualize the strongest topic-pair overlaps as networks and full pairwise matrices, respectively.The networks show edge width proportional to shared mass, while the matrices set diagonal entries to zero.
  • Findings: MonoTM yields qualitatively different topic relations and explains them with semantic features such as clinical disease investigation and engineering optimization.The shared feature vocabulary supports topic description, comparison, and inspection of attributes crossing benchmark category boundaries.
Loading 2609.09575v1…