PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Syntax Matter? A Graph-Augmented Variational Topic Model for Computational Social Sciences

Alessandro Meneghini

arXiv:2609.07797v1cs.CL

TL;DR

Topic modeling can trade interpretive depth for breadth when analyzing large text corpora. The paper proposes SCPTM, combining syntactic graph encoding with probabilistic mixed-membership topic inference, and evaluates when syntax adds value. SCPTM-none achieved an NMI of 0.391 against CTM’s 0.232, a 68% improvement in topic-label alignment.

  • Problem

    Topic modeling can trade interpretive depth for breadth when analyzing large text corpora.

    Source: §1 · p. 2

  • Method

    The paper proposes SCPTM, combining syntactic graph encoding with probabilistic mixed-membership topic inference, and evaluates when syntax adds value.

    Source: §1 · p. 3, §2.3 · p. 6

  • Results

    SCPTM-none achieved an NMI of 0.391 against CTM’s 0.232, a 68% improvement in topic-label alignment.

    Source: §6 · p. 26, §8 · p. 27

  • Takeaways & Limitations

    The no-graph SCPTM variant's NMI advantage over CTM shows that topic-label alignment gains can occur without graph syntax.

    Source: §6 · p. 26, §8 · p. 27

    The study leaves temporal dynamics for future work, which could track how discourse frames evolve over time.

    Source: §7 · p. 27

Abstract

from arXiv · show

Topic modeling is widely used in computational social sciences to identify latent themes in large text corpora. Traditional approaches rely on Bag-of-Words representations and generative models such as LDA, while recent methods like BERTopic operate on dense document embeddings. This paper introduces the Structural Contextual Probabilistic Topic Model (SCPTM), an architecture that incorporates syntactic dependency relations into topic inference. SCPTM represents a corpus as a heterogeneous graph of documents and words connected by lexical and syntactic edges, processed through a Graph Attention Network within a Variational Autoencoder to produce probabilistic, mixed-membership topic distributions. We evaluate seven topic modeling techniques (including four SCPTM ablations) across four corpora differing in register and discourse structure. Our framework combines coherence (C_V, C_NPMI), topic diversity, clustering-label alignment (NMI), and phrase-level diagnostics (complementarity and valence gap). Results show that SCPTM's neural architecture yields substantial gains in document-topic alignment over generative baselines, but these gains are attributable to the variational encoder rather than to syntax. Syntax contributes to topic diversity, where graph-augmented variants outperform the no-graph baseline across all corpora, and to descriptor quality: dependency paths capture predicate-argument structures and stance in deliberative registers, while proving redundant in technical and institutional corpora. The valence gap is positive across all variants, but driven primarily by phrase grouping rather than syntactic filtering. We conclude that syntactic encoding matters conditionally: it benefits action-oriented, argumentative texts, but introduces noise in informational or administrative registers.

1. Introduction

The introduction asks whether syntactic encoding improves topic quality, and under what corpus conditions. It presents SCPTM and a comparative evaluation spanning models, corpora, and multiple measures of topic quality.

1. Introduction

  • SCPTM combines syntactic graph encoding with probabilistic mixed-membership topic inference to test when syntax improves topic quality.Its evaluation is designed to assess whether, where, and under what conditions syntactic encoding yields measurable benefits.Source: §1 · p. 3
  • Seven models, including four SCPTM graph-composition ablations, are evaluated across four corpora that vary in register, formality, and discourse structure.The corpora are 20 Newsgroups, European Parliament Debates, Reddit political discussions, and hate speech.Source: §1 · p. 3
  • The evaluation combines coherence, topic diversity, intra-topic concentration, and NMI because these metrics measure distinct properties of topic quality.The measures cover semantic coherence, lexical spread, document clustering, and correspondence to external categories.Source: §1 · p. 3
  • Multi-word-expression diagnostics assess complementarity and valence gap for syntactically based collocations relative to unigram baselines.These diagnostics extend the comparison to whether syntax adds value at the phrase level.Source: §1 · p. 3

2. What Is a Topic? A Matter of Representation

The section contrasts topics as word-probability distributions with topics identified through embeddings, then presents SCPTM as a structural alternative. It motivates each representation by what it preserves and what it leaves out.

2.1 Topic as Probability Distribution: Generative Models

  • Bag-of-Words topic models treat documents as token sets, so sentences with the same vocabulary but different action roles can receive similar representations.They also miss word-sense distinctions and synonymy, treating vocabulary items independently.Source: §2.1 · p. 4

2.2 The Word Embeddings Perspective

  • Embedding-based topic models place documents in semantic space, where clustering identifies topics; transformer embeddings can improve coherence and distinguish senses.Hard clustering may not fit mixed-membership texts, and frequency-based keyword extraction may not preserve the input's semantic richness.Source: §2.2 · pp. 4–5

2.3 Topics as Structural Encoding of Semantics: The SCPTM Architecture

  • SCPTM defines a topic as a recurring pattern of syntactic dependencies, rather than solely a word list or a position in semantic space.It represents documents and words in a graph with lexical and syntactic edges, then uses a GAT and VAE to infer probabilistic mixed-membership topics; topics include unigrams and dependency-grounded MWEs.Source: §2.3 · pp. 5–6
  • Preserving dependency structure lets SCPTM distinguish documents that share vocabulary but use it in systematically different syntactic configurations.For example, the representation of “border” changes with its role, such as object, modifier, or compound.Source: §2.3 · pp. 5–6

3. Related Work

Related work includes syntactic extensions of probabilistic topic models, Bag-of-Words models, embedding-based approaches, and graph methods. SCPTM is positioned as defining topics through recurring syntactic dependencies while combining structural information with mixed-membership inference.

3. Related Work

  • SCPTM defines a topic as a recurring pattern of syntactic dependencies appearing across documents.The definition motivates a model that operates on syntactic structure, preserves mixed membership, and captures semantic similarity.Source: §3 · p. 6

3.1 Syntactic extensions of topic models

  • Earlier syntactic topic models incorporate grammatical information into generative Bag-of-Words approaches, including dependency structure in the word-generation process.The cited approaches include bigram-based extensions and Syntactic Topic Models.Source: §3.1 · p. 7

3.2 Bag-of-words approaches

  • Bag-of-Words models retain mixed membership, with CTM and STM adding topic correlations and covariates, but cannot distinguish a word's different syntactic uses.Their mixed-membership representation can capture documents addressing multiple topics with different weights.Source: §3.2 · p. 7

3.3 Embedding-based approaches

  • Embedding-based approaches represent semantic similarity geometrically or cluster document embeddings, with topic descriptions often based on frequency metrics such as c-TF-IDF.CombinedTM uses contextual SBERT embeddings in a VAE; BERTopic and Top2Vec cluster documents after dimensionality reduction.Source: §3.3 · p. 7

3.4 Graph-based approaches

  • Graph approaches use lexical co-occurrence networks or document-word graphs to propagate information through shared words and stabilize topic extraction.TextGCN uses PMI-weighted document-word graphs for classification, while lexical networks can make topics less sensitive to frequency skew and initialization.Source: §3.4 · pp. 7–8

4. Structure and Semantics: the SCPTM Model

This section describes SCPTM’s approach to mapping documents to probabilistic topic mixtures using semantic embeddings and syntactic dependency structure, along with its training objectives and extraction of single-word and syntax-based multi-word descriptors.

4. Structure and Semantics: the SCPTM Model

  • SCPTM embeds words and documents, encodes their semantic and syntactic information with graph attention and a variational autoencoder, then extracts single-word and dependency-path descriptors.Source: §4 · p. 8

4.1 Semantic Initialization

  • SBERT document vectors represent contextual meaning, while word vectors average a lemma’s occurrences; k-means word-embedding centroids initialize learnable topic embeddings that training reshapes.Reconstruction pulls topic centroids toward co-occurring words, while a diversity penalty pushes overly similar centroids apart.Source: §4.1 · pp. 9–10

4.2 Syntactic Graph Construction

  • The heterogeneous graph links documents to words they contain and links words through selected dependency relations, so shared vocabulary in different syntactic configurations can yield different topic assignments.Source: §4.2 · p. 10

4.3 Graph Attention Encoder

  • The Graph Attention Network (Veličković et al., 2018) weights neighboring nodes by context, but pools dependency labels into one relation type and cannot learn relation-specific attention weights.The paper identifies relation-specific attention weights as a direction for future work.Source: §4.3 · p. 11

4.4 From Document Representation to Topic Distribution

  • A variational autoencoder (Kingma & Welling, 2022) maps each document representation to a continuous probability distribution over topics, yielding mixed-membership assignments and a regularized latent space.Source: §4.4 · p. 12
  • The posterior q(z|d) = N(µ_d, σ_d^2 I) is sampled and passed through softmax to produce θ_d, the document’s topic proportions.The model interprets θ_d as positive values summing to one that represent the proportions with which a document addresses each topic.Source: §4.4 · pp. 12–13
  • The ELBO balances reconstruction of document word counts from θ_d against a penalty for deviating from the standard Gaussian prior.The prior regularizes the posterior to encourage generalization across documents.Source: §4.4 · p. 13
  • A topic-diversity penalty encourages distinct topic embeddings and discourages multiple topics from collapsing into the same semantic region.Source: §4.4 · p. 13

4.5 Inference and Topic Extraction

  • At inference, SCPTM obtains θ_d without sampling and ranks vocabulary terms by cosine similarity to each topic embedding for single-word keywords.The top-n terms are those whose SBERT embeddings are geometrically closest to the topic centroid.Source: §4.5 · p. 13
  • SCPTM also extracts multi-word descriptors from directed length-2 paths in the syntactic graph, scoring candidates by mean embedding similarity and retaining the top 20 per topic.Source: §4.5 · pp. 13–14

5. Empirical Evaluation

The evaluation compares seven topic-modeling techniques across four corpora selected to vary in register and discourse structure. Its design tests whether lexical and syntactic graph edges contribute to topic quality across complementary metrics.

5. Empirical Evaluation

  • Seven models, including four SCPTM configurations, are compared with LDA, CombinedTM, and BERTopic across four corpora.The SCPTM configurations are designed to isolate the contributions of lexical and syntactic graph edges.Source: §5 · p. 14
  • SCPTM adds selected dependency edges to document–word edges, while SCPTM-fulldep includes all content-word dependency types and excludes function-word relations.SCPTM-fulldep therefore isolates the effect of restricting the syntactic relation types.Source: §5 · p. 14
  • The evaluation triangulates coherence, topic diversity, label alignment, and phrase-level complementarity and valence gap rather than relying on one score.The corpora also differ in whether external category labels are available.Source: §5 · p. 14

5.1 Corpora Selection

  • The corpora span differing registers and discourse structures, allowing the evaluation to test whether syntactic structure helps conditionally rather than universally.They include 20 Newsgroups, EU Parliament Debates, Reddit Politics, and Measuring Hate Speech.Source: §5.1 · p. 15

5.2 Evaluation Metrics

The evaluation uses multiple diagnostics because topic quality has no single decisive score. SCPTM improves clustering alignment over generative baselines, but coherence leaders vary by corpus and its diversity declines as topic count grows.

5.2 Evaluation Metrics

  • The evaluation combines C_V and C_NPMI coherence, topic diversity, NMI alignment where labels exist, and phrase-level complementarity and valence gap.The stated rationale is to triangulate partially conflicting signals rather than optimize a single score.Source: §5.2 · pp. 15–16

5.3 Baseline Comparison: Neural Architecture vs. Generative Models

  • The SCPTM family surpasses CTM in clustering quality across all three labeled corpora, while alignment gains are attributed to the variational encoder rather than syntax.Table 1 reports topic-quality metrics by model and corpus at each model’s best K, averaged over three seeds.Source: Table 1 · p. 16, §5.3 · p. 17
  • SCPTM-none achieves the highest NMI on two labeled corpora: 0.3911 on 20 Newsgroups and 0.2231 on Hate Speech.The passage reports that adding lexical or syntactic edges has mixed effects on alignment.Source: §5.3 · p. 17
  • BERTopic leads C_V on 20 Newsgroups (0.5842), EU Debates (0.6094), and Reddit (0.4633), while CTM leads on Hate Speech (0.5773).BERTopic is also the only model with positive C_NPMI across all corpora, ranging from 0.0327 to 0.1496.Source: §5.3 · p. 17
  • SCPTM models display lower coherence scores, with C_V ≈ 0.37–0.43 and C_NPMI around −0.21 to −0.31; across SCPTM variants, coherence remains relatively stable.Source: §5.3 · p. 17
  • SCPTM diversity is 0.69–0.78 at best K, but drops to 0.53 at K=30; CTM’s diversity is 0.92–0.98 at best K.The reported SCPTM decline accompanies increasing topic counts.Source: §5.3 · pp. 17–18

5.4 Syntactic Diagnostics: MWE Metrics and Valence Gap

Phrase-level diagnostics find a positive valence gap across all corpora and SCPTM variants, but the gap is attributed mainly to phrase grouping, not syntactic filtering. The graph’s contribution is structural, constraining descriptors to grammatical relations.

5.4 Syntactic Diagnostics: MWE Metrics and Valence Gap

  • The valence gap is positive across all four corpora and model variants at each best-K configuration, indicating greater evaluative charge in multi-word descriptors than top unigrams.The metric compares evaluative degree in syntactic co-occurrences with individual lexical choices.Source: §5.4 · p. 18
  • The positive gap is attributed primarily to multi-word grouping, not syntactic dependencies; on 20 Newsgroups, SCPTM-nosyn’s Δ=0.069 exceeds the full model’s Δ=0.063.The reported base-architecture gap is Δ=0.052.Source: §5.4 · p. 19
  • The graph’s role is structural: it filters function words and constrains descriptors to valid grammatical relations rather than increasing sentiment intensity.Source: §5.4 · p. 19
  • Complementarity remains between 0.287 and 0.390 across variants and corpora; phrases share 60% to 70% of their vocabulary with unigrams.The positive Δ is attributed to pairing words already in the topic’s core vocabulary.Source: §5.4 · p. 19

5.5 Does the Graph Help? An Internal Ablation

The ablation compares graph-augmented SCPTM variants with a no-graph baseline to test the contributions of lexical and syntactic edges. Graph structure improves topic diversity, while its effects on clustering alignment and valence gap vary by corpus.

5.5 Does the Graph Help? An Internal Ablation

  • 0.5344 (SCPTM-nosyn) falls to 0.4067 (SCPTM-none) in topic diversity; similar drops occur in Hate Speech and Reddit Politics.The reported comparisons are 0.6922 to 0.6344 for Hate Speech and 0.7778 to 0.6711 for Reddit Politics.Source: Table 3 · p. 19, §5.5 · p. 20
  • On 20 Newsgroups and Hate Speech, SCPTM-none has the highest NMI, while both SCPTM and SCPTM-fulldep beat the no-graph baseline on Reddit Politics.The text attributes the modest alignment declines in the first two corpora to graph propagation smoothing document representations.Source: Table 3 · p. 19, §5.5 · p. 20
  • The diversity gains are strongest in corpora with stable vocabularies, whereas syntactic constraints may add structural noise in factual, operational settings.The authors qualify this interpretation because the sample of corpora is too small for definitive conclusions.Source: §5.5 · pp. 20–21
  • The valence gap stays positive across variants, but full dependency parsing yields the highest gap on Hate Speech and Reddit Politics and a lower score on 20 Newsgroups.The reported gaps are 0.1202 and 0.0439 for the first two corpora, versus 0.0626 on 20 Newsgroups.Source: §5.5 · pp. 20–21

5.6 Qualitative Observations and Representational Patterns

The qualitative comparison examines model descriptors across Reddit Politics, 20 Newsgroups, and European Parliament Debates. It finds that dependency paths can capture action and stance in argumentative text, but often add little in informational or institutional registers.

5.6 Qualitative Observations and Representational Patterns

  • The qualitative comparison inspects descriptors for shared topics across models, using phrase specificity, valence gap, and syntactic redundancy across three registers.The examples are drawn from Reddit Politics, 20 Newsgroups, and European Parliament Debates.Source: §5.6 · p. 21

5.6.1 Action and Stance vs. Static Semantic Clouds (Reddit Politics)

  • Reddit dependency paths capture predicate-argument structures and evaluative stance, unlike LDA’s broad conversational terms or BERTopic’s static unigram clouds.The comparison includes descriptors such as “attack ukraine” and “spread fake news”; whether they improve interpretation requires dedicated human evaluation.Source: §5.6.1 · pp. 21–22

5.6.2 The Limits of Syntax on Factual and Technical Domains (20 Newsgroups)

  • In 20 Newsgroups, dependency-filtered phrases add little beyond unigrams or standard n-gram preprocessing, and can introduce structural noise in informational text.The outputs include generic pairings such as “program send” alongside genuine domain compounds; the authors link this pattern to the syntactic ablation’s stronger topic quality.Source: §5.6.2 · pp. 23–24

5.6.3 Syntactic Redundancy in Institutional Registers (EU Debates)

  • In European Parliament Debates, dependency paths do not systematically disambiguate topics beyond unigrams, and their collocations can be topically redundant or procedural.The authors describe these observations as preliminary and call for blinded human evaluation of any interpretive advantage over bag-of-words or embedding clusters.Source: §5.6.3 · p. 24

6. Discussion

The discussion argues that syntax helps conditionally: dependency structure is useful in deliberative registers but can be redundant in formal or factual ones. It also distinguishes neural alignment gains from syntax’s contributions to diversity and descriptor structure.

6. Discussion

  • Dependency extraction is most effective in deliberative, informal, evaluatively dense registers, while technical and institutional syntax can act as scaffolding and add structural noise.The discussion characterizes bag-of-words as an efficient heuristic in those formal or strictly factual settings.Source: §6 · p. 25
  • C_NPMI and C_V operationalize different properties, so the discussion recommends choosing or reporting them according to the intended theoretical emphasis.C_NPMI tracks document-level co-occurrence stability, whereas C_V is calibrated against human judgments of semantic relatedness.Source: §6 · pp. 25–26
  • SCPTM retains each document’s continuous probability distribution over topics rather than forcing it into one discrete category.The discussion frames this mixed-membership representation as suited to texts with intersecting discourses and competing frames.Source: §6 · p. 26
  • SCPTM-none achieved NMI 0.391 versus CTM’s 0.232, a 68% topic-label alignment improvement attributed to the variational encoder and semantic initialization, not the syntactic graph.Adding dependency edges did not further improve NMI on this corpus and sometimes slightly reduced it.Source: §6 · p. 26

7. Limitations and Future Work

The section notes computational and interpretive costs, English-only empirical benchmarks, and dependence on language-specific parsers. It also identifies unvalidated phrase descriptors and several directions for extending SCPTM.

7. Limitations and Future Work

  • Neural topic models inherently demand more computational resources than frequency-based clustering, while work on massive corpora (>100,000 documents) must budget appropriate upfront preprocessing time.The reported study comprised 504 model fits across seven architectures, four corpora, three random seeds, and six K values, and took approximately 11.5 wall-clock hours.Source: §7 · p. 26
  • The benchmarks are restricted to English, and applying SCPTM depends on the availability and quality of language-specific dependency models.The section identifies low-resource languages without robust parsers as a challenge; SCPTM-none remains applicable in those cases.Source: §7 · p. 27
  • Syntactic descriptors can require active interpretation rather than offering immediate keyword-level legibility, and their substantive utility lacks human evaluation.The authors identify a formal, blinded annotation study as a next step.Source: §7 · p. 27
  • Future work includes modeling temporal change, using multi-word valence density in downstream tasks, and developing researcher-guided topic initialization.The section presents these as objectives for moving SCPTM toward confirmatory analysis.Source: §7 · p. 27

8. Conclusion

The conclusion presents SCPTM as a mixed-membership topic model combining contextual embeddings with dependency-graph attention. It reports alignment gains from the neural architecture and conditional benefits of syntax, which depend on register.

8. Conclusion

  • SCPTM preserves mixed-membership logic by assigning documents probability distributions over topics, rather than treating each text as a single category.The conclusion describes the architecture as applying graph attention over dependency-parsed edges while integrating contextual embeddings.Source: §8 · p. 27
  • On 20 Newsgroups, SCPTM-none achieved NMI 0.391 versus CTM’s 0.232, while syntactic edges added topic-diversity gains and qualitatively distinct descriptors.The conclusion attributes alignment advantages to neural architectures and describes dependency paths as capturing predicate-argument structures and stance.Source: §8 · pp. 27–28
  • The valence gap between multi-word and unigram descriptors is positive across variants, but the effect is driven primarily by phrase grouping rather than syntactic filtering..Source: §8 · p. 28
  • The authors argue that topic models should be assessed with complementary measures that capture different epistemic dimensions of text..Source: §8 · p. 28
  • Syntax is useful in deliberative, argumentative, or evaluatively dense registers, where graph encoding captures predicate-argument structures and stance.In strictly factual, technical, or institutional corpora, grammatical constraints may be redundant and cleaner semantic isolation may come from unconstrained neural baselines or density-based clustering.Source: §8 · p. 28
  • The conditions under which syntax helps remain incompletely understood, and whether diversity gains improve human interpretability remains untested.The conclusion also says the sources of the positive valence gap require further disentanglement.Source: §8 · p. 28

Related papers

Loading 2609.07797v1…