Source-linked AI summary
Recovering Temporal and Geographic Signals from Language Model Embeddings
Esteban Feuerstein, Victoria Klimkowski, Juan Manuel Ortiz de Zarate, Federico Hernán Suaiter
TL;DR
The paper asks whether language-model output embeddings contain structured temporal and geographic information in a form that can be analyzed without model internals or additional training. It introduces a projection-based, black-box method using seed examples to define embedding axes and rank texts or entities, finding meaningful temporal and spatial structure with potential interpretability and information-retrieval applications.
Problem
The paper addresses the need to determine whether output embeddings encode structured temporal and geographic information without relying on internal activations, model weights, probes, or task-specific supervision.
Method
The method selects seed examples, defines projection-based temporal or geographic dimensions in embedding space, and ranks texts or entities using only model-produced embeddings.
Results
The projections recover meaningful chronological and spatial structure across temporal and geographic datasets, including a Kendall’s τ score of 0.87 for voyage-lite-01 on Conservative Speeches.
Takeaways & Limitations
The approach provides a lightweight interpretability tool and may support unsupervised temporal or geographic tagging, ordering, ranking, and information retrieval.
Takeaways & Limitations
The representation is local: the method works effectively as an interpolator but struggles to extrapolate.
Abstract
from arXiv · showhide
Understanding whether language-model embeddings encode structured real-world information is important for both representation analysis and information retrieval. We study this question for temporal and geographic signals using a simple projection-based method that operates directly on output embeddings. Given a small set of seed examples, the method defines an axis in embedding space and ranks texts or entities by their projection onto that axis. Our approach is fully black-box and model-agnostic: it requires only embeddings, without access to model weights, internal activations, auxiliary probes, or additional training. This makes it applicable to modern embedding models available only through APIs and provides a lightweight way to analyze whether temporal and spatial dimensions are present in their representation spaces. We apply the method to temporal and geographic datasets and find that embedding projections recover meaningful chronological and spatial structure. These results provide evidence that output embeddings encode signals relevant to time and space, while also offering a practical tool for interpretability and for downstream temporal and geographic information retrieval tasks, such as temporal ordering, geographic ranking, and tagging.
1 Introduction
This work examines whether language-model embeddings encode grounded temporal and geographic structure, extending prior analyses of internal representations with a simple embedding-only method. The approach supports interpretability and potential temporal and geographic information-retrieval applications.
- Prior work found that LLMs can learn linear representations of space and time that remain robust across prompt variations and entity types.Related analyses also identified individual neurons encoding spatial and temporal coordinates.
- The proposed method uses output embeddings to define temporal and spatial dimensions without accessing internal network layers.It maps corpus texts onto a temporal dimension and evaluates the resulting ordering, then applies a similar procedure to geographic structure.
- The study investigates temporal ordering, temporal granularity, alternative signal-construction strategies, and the challenges of representing three-dimensional geographic information.The temporal analysis tests how accurately projections order texts and how the signal behaves at different granularities.
- Unlike prior approaches requiring internal activations, model weights, probes, or task-specific training, this black-box method requires only language-model embeddings.It is intended for modern embedding models exposed only through APIs and frames temporal text classification as a ranking or ordering problem.
- The resulting analysis offers a lightweight interpretability tool and may support temporal and geographic information-retrieval tasks.Potential applications include temporal ordering, geographic ranking, and downstream retrieval workflows.
2 Related Work
Related work establishes that language and embedding models capture syntactic, semantic, temporal, and geographic regularities, but existing approaches often rely on supervision, internal representations, or limited spatial reasoning. This paper applies d-ness scoring directly to text embeddings to recover temporal and geographic dimensions.
- Sentence and word embeddings represent semantic relationships using vector operations and cosine similarity, often without explicit supervision.SBERT obtains sentence embeddings through pooling and fine-tuning so they can be compared efficiently using cosine similarity.
- TempoBERT improves temporal awareness through time-masking and task-specific temporal supervision, unlike the present off-the-shelf embedding setting.Earlier work also used lexical features and language models for automatic document dating.
- Prior geographic studies report some spatial reasoning but limited geographic information, with larger models generally performing better.Tested models were not reliably able to reason about cardinal directions.
- The paper uses d-ness scoring to analyze model-output embeddings rather than internal weights, extending a projection principle previously applied to other embedding objects.The claimed novelty is applying d-ness scoring to temporal and geographic dimensions in text embeddings.
3 Method
The method constructs temporal and geospatial dimensions from embedding vectors and ranks documents by projection onto those dimensions. It uses multiple seed-selection strategies and ranking correlations to test signal robustness across models and datasets.
- The approach hypothesizes that textual events, time expressions, topics, locations, monuments, and cities induce temporal and geospatial regularities in embedding space.
- A dimension is initialized from seed pairs intended to differ primarily along the target dimension, then documents are projected onto it using cosine similarity.The method can be generalized to multiple seed pairs for a more robust dimension.
- Using multiple seed pairs averages similarity to the first components against similarity to the second components, so high d-ness indicates greater alignment with the first set.This interpretation relies on choosing paired examples that are otherwise similar across dimensions.
- Seed selection strategies: Three temporal seed strategies test robustness: extremes capture global change, total random provides a near-zero-correlation control, and ordered random tests whether chronology extends beyond absolute extremes.
- Temporal analysis: Chunking partitions the timeline into contiguous segments and varies the number of segments to examine how temporal-window length affects signal stability.Each chunk applies the seed-sampling strategies with K seeds.
- Evaluation: The resulting scores define rankings evaluated with Kendall’s τ and Pearson’s r, enabling comparisons that are less sensitive to differences in score scale.Kendall’s τ ranges from −1 for disagreement to 1 for agreement, with 0 indicating no correlation.
- Embedding models: Experiments use black-box embedding endpoints, including OpenAI ada-002 and Google Gemini text-embedding-004, alongside additional models for high-density analysis.
4 Experimental Results
The experiments evaluate whether embedding projections recover temporal and geographic structure across datasets, models, seed strategies, and hyperparameters. Results show strong local temporal ordering and meaningful spatial correlations, while also revealing limits on temporal extrapolation and geographic accuracy.
- Experimental setup: The experiments assess temporal and geographic dimensions across multiple embedding models and datasets.The setup includes presidential speeches, high-density news, and geographic place datasets, with varying seed-selection strategies and hyperparameters.
- Temporal results: τ values above 0.8 were achieved by voyage-lite-01 on presidential-speech rankings, with correlation increasing as K increased across both models.The comparison included voyage-lite-01 and ada-002, with voyage-lite-01 generally producing stronger temporal rankings.
- Temporal results: Temporal rankings remained robust on high-density daily news, and Random Sorted produced better correlations for larger K values.The results indicate that temporal structure persists beyond widely separated extreme dates and can smooth daily topical noise.
- Temporal limits: Local temporal axes yielded better precision than global axes, while projections interpolated between seeds but failed to reliably extrapolate beyond them.The extracted temporal dimension is tied to event progression in a local period rather than representing a universal absolute time magnitude.
- Geographic limits: Geographic predictions were accurate for some landmarks but substantially displaced for the Statue of Liberty, potentially because of replicas and ambiguous common words.The reported discrepancy illustrates that geographic signals can be affected by how locations are represented in language.
- Geographic results: For PopulatedPlaces, the strongest longitude segment reached τ=0.848 and r=0.931, suggesting that the model may capture longitude’s circular representation.Other segments were weaker, with τ=0.788, r=0.79 and τ=0.451, r=0.7.
5 Discussion and Further Work
The projection method offers a lightweight way to recover temporal and geographic structure from embeddings and potentially support retrieval-oriented ranking and tagging. The discussion also identifies important scope boundaries: temporal representations are local, extrapolation is difficult, and the causes of recovered signals remain uncertain.
- The projection procedure operates solely on embeddings and can probe grounded temporal and geospatial structure for interpretability and information retrieval.The paper suggests applications including unsupervised temporal or geographic tagging and dimension-aware ranking.
- Temporal findings: Temporal signals persist across yearly and daily datasets, including when seed selection avoids absolute temporal extremes.Larger temporal windows also smooth short-term semantic noise and improve linear ordering relative to smaller fragmented windows.
- Temporal limitations: The temporal representation is local rather than universal: the method interpolates effectively but struggles to extrapolate.
- Geographic findings: Spatial projections correlate with geographic locations, while longitude appears nonlinear and may require circular rather than linear projection.
- Interpretation and future work: The source of temporal signals is unresolved because topical drift, jargon, grammatical variation, semantic domain, and supervision may all contribute.The authors explicitly avoid claiming a unique explanation and report additional experiments comparing broader and narrower context.
- Further work: Further work includes validating additional domains, improving temporal fine-tuning, and studying how prompt formulations affect signal emergence.Additional experiments on a symbolic music language model reportedly found relevant signals, but those results are not included in the article.