When Can We Work in Embedding Space? What Text Embeddings Preserve

• The paper makes precise when text embeddings can serve as inputs to empirical analysis by deriving what they preserve under a generative model of documents as mixtures of latent topics.This addresses the possibility that reducing text to a vector discards information needed for analysis.Source: §1 · p. 2
• Under the latent-topic model, word embeddings inherit topic-loading geometry, while SGNS recovers that geometry up to distortion and averaged document embeddings are an invertible linear image of topic mixtures.The centered word co-occurrence matrix has rank K −1, and document embeddings lie in the simplex spanned by topic centroids.Source: §1 · pp. 2, 4
• Embedding-space clustering groups documents with similar topic mixtures, with exact recovery guaranteed when documents concentrate around a few common mixture-types, though separation alone does not ensure a unique k-means optimum.A cluster centroid corresponds to a well-defined mixture.Source: §1 · pp. 2–3
• Controlling for an embedding is equivalent to controlling for the latent topic mixture, so validity requires that the topic mixture capture the confounding.This reduces the assumption that an embedding is a sufficient control to a transparent condition on the underlying topic mixture.Source: §1 · p. 3
• The paper differs from prior work by taking the topic model as the data-generating process and asking what standard unsupervised embeddings recover, complementing studies that treat the recovered target as given.The framework builds on topic-model research and contrasts with supervised confounding embeddings and downstream corrections for imperfect representations.Source: §1 · pp. 3–4

2.1 Generative Model

• The corpus model treats each document’s words as conditionally i.i.d. draws from a topic mixture, with word probabilities Π = BΘ and independent documents conditional on B and Θ.A Dirichlet prior yields standard LDA, while fixed topic mixtures connect the model to NMF.Source: §2.1 · pp. 4–5

2.2 Word Embeddings

• Any embedding that factorizes the centered probability ratio recovers the Σ_Θ-weighted geometry of marginal-normalized topic loadings, so proportional topic loadings receive identical embeddings.The centered probability ratio is PSD of rank K−1, and the geometry is invariant to valid topic-factorization reparameterizations; dimensions r ≥ K−1 preserve it.Source: §2.2 · pp. 8–9

2.3 Relationship to Standard Algorithms

• SGNS targets recover the same topic-loading geometry up to distortion at most κ_R cond(β) cond(Ẇ), whereas trained embeddings depend on how well their weighted low-rank fit approximates that target.The result requires positive word-pair co-occurrence; full-softmax skip-gram and GloVe generally retain row offsets, so the SGNS proposition does not apply unchanged.Source: §2.3 · pp. 10–12

2.4 Document Embeddings

• Expected document embeddings formed by averaging word embeddings are linear functions of the document’s topic mixture under the identified model.Averaging is natural because tokens are conditionally i.i.d.; finite-document averages converge to their expected embeddings as document length grows.Source: §2.4 · pp. 12–13
• Document embeddings lie in the simplex spanned by topic centroids and, under the factorization condition, provide an invertible linear encoding of topic mixtures.Thus nearby embeddings correspond to similar mixtures, and centroid positions map to well-defined topic mixtures.Source: §2.4 · pp. 14–15

2.5 Grouping: Clustering Documents

• For concentrated documents, dominant-topic mixtures fall in the corresponding centroid Voronoi cells, with the assignment margin determined by the ratio of minimum to maximum centroid distances.The guarantee is strongest when centroids are equidistant, where η = 1/2, and weakens as some centroids become relatively close.Source: §2.5 · pp. 16–17
• Under archetype concentration and separation, k-means has the intended partition as a fixed point, but the application does not satisfy these conditions and claims only mixture-type similarity.The guarantee concerns stability rather than global recovery; embedding clusters represent similar topic mixtures without requiring concentration assumptions.Source: §2.5 · pp. 18–19

2.6 Conditioning: Embeddings as Controls

• Embedding adjustment is valid if and only if topic-mixture adjustment is, provided the embedding map is injective; validity requires topic unconfoundedness.The result is exact at the population level for expected embeddings and identifies the average treatment effect under the stated positivity condition.Source: §2.6 · p. 20
• Embedding-based control additionally requires dimension r ≥ K −1 and an injective map; dimension alone cannot ensure recovery of the confounding summary.With r < K −1, the embedding captures only a projection and leaves residual confounding, while the embedding geometry must coincide with the latent summary’s geometry.Source: §2.6 · pp. 20–21

Simulation Design

• The simulations use two topic designs with D = 363 documents, Nd = 487 words per document, and r = K, where the extra SVD-β dimension is population slack.The designs draw topic mixtures from a symmetric Dirichlet α = 1 prior and use a common vocabulary of 16 fruits and vegetables.Source: §3 · pp. 21–22

Design 1: Two Topics

• In Design 1, SGNS and SVD-β produce nearly identical one-dimensional geometries that separate fruits from vegetables and place mixed words near the boundary.Anchor words lie at cluster extremes, while mixed-word positions reflect relative topic loadings; SVD-β matches this pattern exactly and SGNS approximately.Source: §3 · pp. 22–23, Figure 3 · p. 23

Design 2: Three Topics

• Both methods clearly separate the three topics in Design 2, even though only the vegetable topic has an anchor word.Shared words’ differing relative loadings recover the non-citrus–citrus distinction, while Kale lies on the convex-hull edge.Source: §3 · p. 24, Figure 4 · p. 25

Document Embeddings

• Document embeddings fill the topic simplex, and SGNS and SVD-β yield similar geometries when documents’ topic mixtures are spread under α = 1.This follows the prediction that document embeddings lie in the (K −1)-dimensional simplex spanned by topic centroids.Source: Document Embeddings · p. 25, Figure 5 · p. 26

The rank of the SGNS target

• 88% and 91% of spectral mass lies in the leading K −1 eigenvalues for Designs 1 and 2, respectively, making Proposition 2 approximately describe trained SGNS embeddings.Across a broader α = 1 grid, rank-K approximations remain close to the SGNS target, although approximation quality degrades for smaller α.Source: Document Embeddings · p. 26

4 Application: Clustering Metropolitan Economies

• Under the topic-mixture interpretation, k-means groups CBSAs with similar implied mixtures, making cluster centroids interpretable as narrative archetypes.This interpretation follows because embedding distance reflects similarity in document-level topic mixtures.Source: §4 · p. 27
• The pipeline generates 500-word Claude Opus 4.8 descriptions for 363 CBSAs, embeds them with five methods, and applies k-means with L = 5.The methods are SVD-β, corpus-trained SGNS, corpus-trained CBOW, pretrained Google News Word2Vec averaging, and OpenAI text-embedding-3-large.Source: §4 · pp. 27–28
• Embedding clusters recover recognizable metropolitan archetypes, including energy dependence, university and government anchoring, deindustrialization, and diversified growth.The qualitative separation of economic narratives and employment dynamics holds across alternative numbers of clusters.Source: §4 · pp. 28–29

4.1 Local Employment Dynamics

• Employment-dynamics comparisons use AR(2) models with cluster-specific slopes and evaluate between-group impulse-response variance against chance and non-text benchmarks.The analysis is entirely in-sample, and descriptions span 1979–2019 concurrently with the employment outcome window.Source: §4.1 · pp. 29, 31, 32
• Text-based groupings separate employment dynamics more sharply than residual and observables benchmarks, with SVD-β, SGNS, and LLM embeddings generally performing best.At L = 5, every grouping exceeds chance, while the five text-based methods tend to outperform both non-text benchmarks across cluster counts.Source: §4.1 · pp. 31–32, Table 4 · p. 31

Theory

• Document proximity in embedding space reflects proximity in latent topic mixtures, so clustering embeddings groups similar mixtures and controlling for embeddings is equivalent to controlling for topic mixtures.At the word level, co-occurrence-based embeddings recover topic geometry, and extra dimensions do not harm the representation.Source: §5 · p. 32

Assumptions

• Embedding validity depends on two alignments: recovering the text’s low-dimensional structure and recovering the structure the analysis requires.The theory establishes the first alignment, while the second becomes an economic assumption that the topic mixture captures confounding rather than an opaque assumption about embedding dimension.Source: §5 · pp. 32–33

Limitations

• The application suggests that modest gains from later-vintage observables are reassuring because text exceeds both benchmarks by a much larger margin, despite challenges describing smaller CBSAs in the 1960s.This qualifies the empirical comparison by noting a limitation in obtaining historical textual descriptions for some places.Source: §5 · p. 32

Application

• Embedding-based clusters of 363 U.S. metropolitan areas yield interpretable economic types and separate local employment dynamics more sharply than curated industry and demographic covariates.SVD-β and corpus-trained SGNS perform similarly, while pretrained averaging trails at almost every number of clusters; the closed-form embedding matches a commercial transformer at a fraction of the cost.Source: §5 · p. 33

Extensions

• The proposed pipeline preserves large models’ representational power while keeping downstream analysis inspectable and reproducible.It generates text with a language model, embeds it, and then applies the embedding downstream; transformer embeddings and dynamic topic models remain extensions for future theory.Source: §5 · p. 33

Model

• The appendix supplies a common framework for four embedding objectives and derives each algorithm’s population target from the topic model’s pairwise co-occurrences.Section A.6 uses only the fact that every pair of words co-occurs to characterize what embeddings inherit from the topic model.Source: Model · p. 34

A.1 The algorithms as weighted low-rank fits

• Each algorithm represents pair scores as a rank-constrained inner product and fits corpus-derived responses using a convex loss with positive pair weights.The response is fixed by the corpus, whereas the score is the rank-r parameter being estimated.Source: §A.1 · p. 34
• At the population optimum, positive weights do not affect the target: the unconstrained score is X_vu = ϕ′(t_vu), entrywise.Thus the target is determined by the response and convex generator rather than by the weighting scheme, provided the rank constraint is nonbinding.Source: §A.1 · pp. 34–35
• The framework expresses each loss through a Bregman divergence and uses ϕ′ to map real-valued scores onto the response scale.The generators produce squared error, Kullback–Leibler divergence, or Bernoulli log-loss, depending on the algorithm.Source: §A.1 · pp. 34–35

A.2 Full-softmax skip-gram

• Full-softmax skip-gram maximizes population conditional-word likelihood, replacing the corpus-window average in Mikolov et al. (2013c, eqs. 1–2) with population co-occurrence M.The score matrix is X = W ˜W⊤, with X_vu = w_v⊤˜w_u.Source: §A.2 · p. 35
• The unconstrained full-softmax skip-gram target is X* = log R + log q 1⊤ − 1g⊤ for some g ∈ R^V.This is the algorithm’s shifted PMI target under Assumptions 1 and 2.Source: §A.2 · p. 35

A.3 SGNS

• SGNS is a weighted Bernoulli-likelihood fit whose unconstrained target is X* = log R − log ν 1_V 1^⊤, formalizing its weighted matrix-factorization interpretation.This extends Levy and Goldberg’s PMI − log ν characterization by identifying the weights and responses that place SGNS alongside the other algorithms.Source: §A.3 · p. 36

A.4 GloVe

• GloVe’s unconstrained minimizers satisfy X_vu + b_v + b̃_u = log M_vu, with biases absorbing marginal offsets so the interaction target becomes log R_vu.The weighting function f may be any function positive on the support of M.Source: §A.4 · p. 37

A.5 CBOW

• CBOW matches the same log R − log ν target only with single-word contexts; multiword contexts depend on context bundles and lack an analogous V × V PMI factorization.With multiple context words, the target is not a pairwise matrix, leaving the corresponding CBOW factorization open.Source: §A.5 · p. 37

A.6 What the targets inherit from the topic model

• Under the topic model, proportional word loadings imply target rows differing only by row offsets; SGNS has zero offsets, whereas full-softmax and GloVe retain nonzero offsets.The result applies to arbitrary entrywise transformations and offsets constant along rows or columns.Source: §A.6 · pp. 37–38

B.1 Prompts and Generation Settings

• The corpus comprises 363 descriptions of 476–510 words, with a median length of 495; prompts emphasize broad, distinctive economic trends and suppress refusals and boilerplate.These choices aim to avoid invented statistics, ensure coverage of smaller CBSAs, and preserve cross-sectional differences in embeddings.Source: §B.1 · p. 39

B.2 Application Embedding Implementation

• The application embeds identical text preprocessing across five methods and clusters all 363 descriptions using k-means with L = 5, k-means++ seeding, and 500 restarts.The corpus contains 183,633 tokens, 7,411 word types, and near-constant document lengths; four methods average word vectors, while one embeds documents directly.Source: §B.2 · p. 40

B.3 Choice of L for the Application Embeddings

• L = 5 is chosen as a reasonable trade-off because Figure OA.1 shows gradual elbows across methods, consistent with embeddings spread over a simplex rather than concentrated at a few vertices.The plotted quantity is the within-cluster share of embedding variance; UMAP visualizations provide two-dimensional views of the resulting clusters.Source: §B.3 · pp. 42–43, §B.4 · p. 45

B.5 Sensitivity to the Number of Clusters

• Every embedding method remains well above its permutation floor for L ∈ {2, 3, 4, 5, 6, 8, 10}, so employment-dynamics separation is not an artifact of cluster count.Figure OA.3 recomputes the permutation floor at each L to make ratios comparable.Source: §B.5 · p. 46
• Text-based methods outperform the benchmarks at nearly every cluster count, and their advantage over the benchmarks does not depend on L.Residual k-means falls below all five text methods except at L = 8, while observables k-means generally lies between residual k-means and text-based methods.Source: §B.5 · p. 46
• The ordering among the three corpus-based text methods is unstable across the tested cluster-count grid.The compared corpus methods are corpus CBOW, corpus SGNS, and SVD-β across their reported L values.Source: §B.5 · p. 46
• Online Appendix Figure OA.3 plots between-group variance of unit-specific AR(2) impulse responses relative to a size-matched permutation floor for each cluster count.The floor is recomputed at every L, and dashed lines denote the non-text benchmarks.Source: §B.5 · p. 46

Sensitivity to the Embedding Dimension

• Remark 1 predicts that any r ≥ K−1 recovers the same population geometry, so the dimension only needs to be sufficiently large.Because K is unobservable for this corpus, the sensitivity sweep evaluates practical choices r ∈ {10, 25, 50, 100, 200}.Source: §B.6 · pp. 46–47
• SVD-β and corpus SGNS remain flat from r = 10 to r = 200, justifying r = 50 without identifying K.Their between-group variance ratios and cluster balance remain comparably stable across the full range.Source: §B.6 · p. 47
• Corpus CBOW’s between-group ratio shifts by a factor of two because k-means alternates among near-equivalent partition optima.The variation reflects clustering instability rather than a systematic embedding-dimension trend.Source: §B.6 · p. 47

B.7 Sensitivity to the Autoregressive Order

• At p = 2, estimation error accounts for 74% of cross-city IRF dispersion, versus at least 94% at every other order and 100% at p = 5.The benchmark simulates homogeneous common-slope AR(8) data with resampled unit-specific innovations and fits AR(p) models.Source: §B.7 · p. 47
• The analysis therefore treats p = 2 as the best low-dimensional projection, because longer fits spend degrees of freedom and disperse heterogeneity across noisier coefficients despite pooled evidence favoring more lags.The panel contains T = 51 observations per unit.Source: §B.7 · p. 48
• At horizon h = 10 and T = 51, Nickell bias can affect plotted impulse responses at order h/T, and correcting it would likely change levels while increasing estimation noise.The paper leaves split-panel jackknife or analytic correction for future work.Source: §B.7 · p. 48

B.8 Sensitivity to the Context Window

• SVD-β performance falls from 21.8× at the full document to 12.1× at J = 5, whereas corpus CBOW and corpus SGNS exhibit no clear trend.Source: §B.8 · p. 48
• Shorter windows reduce ordered word–context pairs from roughly 80 million to a few million, making SVD-β embeddings noisier when directly factorizing the sample co-occurrence matrix.With vocabulary size V in the low thousands, bR has on the order of six million cells.Source: §B.8 · p. 49

B.9 Dating the Observables: Covariate Vintage

• The observables benchmark’s ratio rises from 15.5× at the 1970 Census vintage to 18.6× at 2000, implying about a 20% look-ahead gain that remains below the text-based grouping margin.The benchmark clusters nine industry, demographic, and education covariates, with population density fixed at 1970.Source: §B.9 · pp. 49–50

B.10 Interpreting the Clusters

• Cluster assignments are reported for all 363 CBSAs across corpus CBOW, corpus SGNS, SVD-β, pretrained Google News vectors, LLM embeddings, residual k-means, and observables k-means.Clusters are numbered 1–5 within each method, with geographic distributions shown separately.Source: §B.10 · p. 51

Cluster Descriptions by Method

• Text-based clustering consistently identifies economic archetypes centered on anchor institutions, amenity migration, regional transition, resource dependence, and severe deindustrialization.Across specifications, cluster descriptions link employment structure to housing shocks, commodity cycles, recession exposure, and shifts from manufacturing toward services.Source: Cluster Descriptions by Method · pp. 61–64
• Alternative text-based specifications vary cluster counts and boundaries but repeatedly describe universities and hospitals, deindustrialization, agriculture, energy, government, tourism, and diversified services.The reported solutions include five clusters with counts such as 25 agricultural CBSAs, 114 eds-and-meds hubs, 101 heavily deindustrialized metros, and 64 diversified service and knowledge economies.Source: Cluster Descriptions by Method · pp. 63–64
• One residual cluster is a singleton and the largest contains 47% of CBSAs because residual k-means minimizes within-group employment-dynamics variation rather than narrative similarity.The method applies k-means with L = 5 to residuals from the pooled AR(2) specification with unit fixed effects.Source: Cluster Descriptions by Method · p. 64
• The residual solution distinguishes government-anchored metros, energy-dependent economies, manufacturing-heavy metros, diversified higher-density areas, and a military outlier.The military singleton, Hinesville–Fort Stewart, has 61.0% military and 79.3% total government employment in 1980, while the government-anchored group contains 47% of the sample.Source: Cluster Descriptions by Method · pp. 64–65
• Residual k-means produces interpretable economic groupings aligned with text-based themes, especially energy and manufacturing archetypes, providing qualitative external validation.The residual clusters include diversified, energy-dependent, government-anchored, manufacturing-heavy, and military-outlier groups.Source: Cluster Descriptions by Method · p. 65
• Online Appendix Figure OA.8 maps cluster assignments across all 363 CBSAs, with cluster numbers comparable within each method but not across panels.Each dot represents a CBSA and colors encode its assigned cluster within the corresponding method.Source: Cluster Descriptions by Method · p. 66

C Additional Numerical Results

• The appendix presents additional numerical illustrations for alternative specifications.It frames the following appendix material as robustness-oriented numerical results rather than a new substantive clustering method.Source: C Additional Numerical Results · p. 67

Large-corpus geometry

• At K = 2, the larger-corpus SVD-β embedding cloud collapses onto a line, matching Proposition 1’s predicted one-dimensional geometry.The comparison increases documents from 363 to 1,000 and document samples from 487 to 10,000; ordered word–context pairs rise from 8.6 × 107 to 1.0 × 1011.Source: §C.1 · p. 67

Large-corpus implementation

• SGNS cannot use full-document context at this scale, so J = 5 yields roughly 6.0×107 pairs per epoch, about 30% fewer than Figure 3.The effective half-window is drawn uniformly on {1, . . . , J}; this implementation change makes the larger-corpus comparison unlike SVD-β’s pure sample-size comparison.Source: §C.1 · p. 67

Large-corpus geometry

• For K = 3, both SVD-β and SGNS separate the three topics in the larger-corpus word embeddings.Each panel is projected onto its leading two principal directions; this avoids an arbitrary oblique slice for SGNS’s optimizer-chosen CBOW coordinates.Source: §C.1 · p. 68
• With α = 1, larger-corpus document embeddings fill the triangle spanned by the three topic centroids, as Proposition 3 predicts.The topic mixtures are spread across the simplex, and embeddings are colored by dominant topic.Source: §C.1 · p. 69

Sparse prior

• At α = 0.01, embeddings concentrate near topic-centroid vertices rather than filling the centroid simplex, because most documents are nearly pure-topic.The simulations hold D = 363 and N_d = 487 fixed while changing only the Dirichlet concentration parameter.Source: §C.2 · p. 70

Boundary regime

• When θ_d,k*(d) ≈1, the dominant-topic threshold θ_d,k*(d) > 1 −η is met for essentially every document, and k-means recovers the dominant-topic partition.Pr(θ_d,k* > 0.5) ≈1 for both K = 2 and K = 3 under α = 0.01.Source: §C.2 · p. 71

Design 1

• For K = 2, document embeddings cluster at the endpoints [c1, c2] rather than filling the line segment.Color indicates the share of topic 1.Source: §C.2 · p. 71

Design 2

• For K = 3, embeddings concentrate at simplex vertices, and both embeddings with k-means recover the well-separated dominant-topic partition cleanly.The vertex concentration contrasts with Figure 5.Source: §C.2 · p. 72

C.3 Interior Archetypes (L < K)

• In the empirical application, k-means recovers L = 5 clusters whose centroids are interior mixtures of an unknown number of latent topics.Thus, the number of resolvable archetypes need not equal the unobserved number of latent topics.Source: §C.3 · pp. 73–74
• The interior-archetype simplex is substantially flatter because the topic-mixture covariance is nearly rank one, and this flattening persists in full coordinates rather than arising from projection.The example draws topic mixtures tightly around interior archetypes using Dirichlet(20 · π∗ℓ) draws.Source: §C.3 · p. 73
• For L = 2 < K = 3, embedding-space cluster identity follows archetype assignment rather than proximity to a dominant-topic simplex vertex.The archetypes are fruit-heavy with some citrus and nearly pure vegetable, and neither lies at a simplex vertex.Source: §C.3 · p. 73

C.4 CBOW

• The appendix reproduces the counterpart figures using continuous bag-of-words embeddings.CBOW is applied to the application-sized corpus, with colors marking the dominant topic in the true B.Source: §C.4 · p. 75

C.5 UMAP Projections

• UMAP is used for application-scale visualization because embeddings range from 50 to 3,072 dimensions, and its nonlinear transformation can warp the simplex while generally preserving clusters.The application uses UMAP instead of retaining the first two coordinates.Source: §C.5 · p. 76
• Cluster structure is preserved under UMAP at both application scale and the larger corpus scale.The comparison covers D = 363, N_d = 487 and D = 1,000, N_d = 10,000, J = 5 designs.Source: §C.5 · p. 76

C.6 Rank of the SGNS Target under a Sparser Prior

• πK−1 falls to 0.68–0.89 at α = 0.1, versus 0.91–0.99 at α = 1, so rank-(K −1) misses a tenth to a third of log R’s spectral mass.The comparison comes from Online Appendix Table OA.3, which averages 8 population draws; the largest standard deviation is 0.016.Source: §C.6 · p. 77
• At α = 0.1, πK−1 is no longer invariant to V for moderately sized K, suggesting that vocabulary size raises the cost of the rank constraint.The sparser prior concentrates topic-word and document-topic distributions more strongly than α = 1.Source: §C.6 · p. 77
• The departure diagnostic ϱ ranges from 0.84 to 6.5 and exceeds one in 25 of 30 cells, so the expansion log R = ββ⊤ + O(ϱ2) under ϱ < 1 does not cover most cases.At α = 1, ϱ remains below 0.6 in every cell, highlighting the stronger dependence departure under the sparse prior.Source: §C.6 · p. 77