Source-linked AI summary
Location-Aware Language Models via Secondary Embeddings
Gokul Srinivasagan, Munir Georges
TL;DR
Pretrained language models inadequately encode geographic semantics for place names and spatial entities. The paper adds structured location signals and targeted masking to pretrained embeddings without tokenizer changes or costly retraining, improving geo-spatial alignment while preserving comparable GLUE performance. The approach also shows consistent benefits across architectures and model scales, with tokenizer choice influencing adaptation effectiveness.
Problem
Pretrained language models often fail to encode geographic semantics and real-world spatial relationships for location entities.
Method
The method augments location names with latitude and longitude through secondary embeddings and applies location-specific masking during lightweight continual pretraining.
Results
Location-aware models preserve comparable GLUE performance while improving geo-spatial alignment, with location-specific masking producing more consistent gains across architectures.
Takeaways & Limitations
Geo-spatial awareness can be added to pretrained language models through lightweight input augmentation without tokenizer modification, vocabulary expansion, or costly retraining.
Takeaways & Limitations
The approach is a post-hoc adaptation focused on encoder-based models, and stronger representations may result from incorporating geographic information during large-scale pretraining.
Abstract
from arXiv · showhide
Pretrained transformer-based language models achieve strong performance across a wide range of NLP tasks but remain limited in encoding geo-locational semantics, leading to suboptimal representations of place names and spatial entities. In this work, we propose a lightweight, model-agnostic approach for injecting geo-spatial awareness into pretrained embeddings without modifying the tokenizer or requiring costly retraining. Our method augments input representations with structured geographic signals by combining location names with their corresponding latitude and longitude, and employs a location-focused masking to better align textual representations with real-world spatial relationships. This design allows the model to incorporate geo-spatial context while preserving existing semantic and syntactic knowledge. Experimental results demonstrate substantial improvements in geo-spatial alignment while maintaining comparable performance on standard NLP benchmarks such as GLUE. The method is computationally efficient, requiring only minutes of additional training, and generalizes across multiple model architectures and scales.
1 Introduction
Pretrained language models capture general linguistic relationships but lack reliable geographic grounding for place names and spatial entities. The paper proposes lightweight structured location signals to improve geo-spatial representations without tokenizer changes or costly retraining.
- Motivation: Pretrained models often place geographically distant cities close together in embedding space, revealing weak geo-spatial awareness.Saarbrücken and Salzburg are cited as an example despite being geographically distant.
- Motivation: Subword tokenization fragments location names and cannot distinguish identically named places such as Paris, France and Paris, Texas.These issues can cause distinct entities to collapse into the same vector-space region.
- Limitations of Existing Approaches: Existing geographic adaptations often require architectural changes, vocabulary expansion, or costly retraining, limiting scalability and practical applicability.Adding 1,000 BERTbase tokens increases embedding parameters by approximately 3.2%.
- Proposed Approach: The proposed method injects structured geographic signals into pretrained embeddings without modifying the tokenizer or requiring large-scale retraining.The approach is designed to preserve existing semantic and syntactic knowledge while adding location awareness.
- Contributions: The method reports substantial geo-spatial improvements without degrading GLUE performance and generalizes across architectures, model scales, and multilingual settings.The reported contribution emphasizes minimal computational overhead and preserved downstream performance.
2 Related Work
Prior methods improve geographic representation through specialized objectives or architectures, whereas this work uses lightweight input augmentation to add geographic signals while preserving general semantic knowledge.
- Prior Approaches: Prior approaches combine masked entity prediction, contrastive learning, or dedicated geographic encoders to model spatial relationships.Examples include SpaBERT, GeoLM, and MGeo.
- This Work: The proposed method adds geographic signals through lightweight input augmentation without tokenizer modifications or additional objectives beyond masked language modeling.It requires only a short training period and aims to preserve general semantic knowledge.
3 Method
The method builds location-enriched training data and represents each entity with its name, latitude, and longitude. It trains pretrained models with standard or location-focused masking to align textual and geographic information.
- 3.1 Dataset Generation: The synthetic corpus contains 820 train-station location names and 5,965 text samples with 62,758 location-entity occurrences.Training descriptions are generated from German long-distance train schedules using Mistral and LLaMA 3.
- 3.2 Continual Pretraining with Secondary Embeddings: The framework inserts location information with < loc >, < /loc >, and < l_sep > tokens to define structured location boundaries.The representation includes the location name and its corresponding coordinates.
- 3.1 Dataset Generation: Each location is associated with latitude and longitude coordinates that form a structured geo-spatial reference for training.The coordinate mapping supplies geographic information alongside textual data.
- 3.2 Continual Pretraining with Secondary Embeddings: Secondary embeddings map each stop_name to latitude and longitude, providing an auxiliary spatial signal beyond standard text-derived token embeddings.The structured format is < loc > stop_name < l_sep > latitude < l_sep > longitude < /loc >.
- 3.2 Continual Pretraining with Secondary Embeddings: Training compares random masking with location masking, which masks stop_name tokens while retaining coordinates for geo-spatial prediction.The targeted objective trains the model to recover locations from contextual and geographic cues.
4 Experiment Details
The experiments evaluate whether geo-spatial information aligns embedding distances with geographic distances while preserving general language performance. They use GLUE, Haversine-based ground truth, correlation metrics, and standard pretrained-model training procedures.
- Evaluation: GLUE evaluation tests whether location-aware pretraining preserves linguistic capability, using task-specific F1, Pearson correlation, or accuracy metrics.Comparable performance to baseline models is treated as evidence that geo-spatial injection does not compromise language understanding.
- Geo-spatial Evaluation: Geographic ground truth is computed with the Haversine formula from latitude and longitude coordinates.Embedding distances are measured with Euclidean distance and cosine similarity.
- Geo-spatial Evaluation: Higher Pearson and Spearman correlations indicate stronger preservation of real-world spatial relationships in the embedding space.The correlations compare geographic-distance and embedding-distance matrices.
- Implementation Details: Models are trained for 25 epochs with batch size 60 and learning rate 1 × 10^-4, while GLUE fine-tuning uses up to 50 epochs and batch size 256.The experiments use AdamW for pretraining and early stopping during GLUE fine-tuning.
- Implementation Details: Location-aware training completes in approximately 15 minutes on a single NVIDIA H100 GPU.This result is reported as evidence of computational efficiency.
5 Results and Discussion
The method improves geo-spatial alignment across several monolingual architectures and scales, while generally preserving language understanding. Gains depend on model size, masking strategy, tokenization, casing, and embedding extraction.
- General Language Understanding: Location-aware models retain GLUE performance comparable to baseline pretrained models across all tasks.BERTbase increases from 81.16 to 81.96 with location masking, and ALBERT increases from 81.44 to 82.07.
- Masking Strategy: Location-specific masking yields more consistent gains than random masking, particularly for BERT and ALBERT.Random masking primarily preserves baseline performance without consistently improving geo-spatial alignment.
- Embedding Analysis: Mean pooling produces stronger geographic alignment than [CLS] extraction, with RoBERTa reaching 0.4503 versus 0.3343 using [CLS].The comparison suggests that geo-spatial information is distributed across tokens rather than concentrated in one representation.
- Representation Factors: WordPiece-based models show more stable improvements than byte-level BPE and SentencePiece models, while uncased models benefit more consistently than cased models.The passages associate these differences with how tokenization and casing represent location names and coordinates.
- Effect of Model Size: BERTlarge achieves the highest geo-spatial correlation at Pearson r = 0.550, while BERTbase and DistilBERT improve over their respective baselines.The results indicate generalization across model scales, although BERT-tiny improves only marginally from 0.0123 to 0.0139.
- Architecture Generalization: RoBERTa correlation increases from 0.2665 to 0.4296, while ALBERT rises from 0.0039 to 0.1746 and MobileBERT from 0.0123 to 0.0345.These results extend the observed gains beyond standard BERT architectures to parameter-efficient and compressed models.
- Multilingual Models: Multilingual mBERT and XLM-R show limited or inconsistent geo-spatial improvement, whereas monolingual and alternative architectures generally improve.The text attributes this challenge to larger, more diverse vocabularies and cross-language variability in entity representations.
6 Conclusion
The proposed lightweight adaptation injects geographic information into pretrained language models without architectural changes, vocabulary expansion, or costly retraining. It improves alignment with real-world geography while maintaining comparable GLUE performance and requires around 15 minutes of training.
- Conclusion: The method augments inputs with structured location signals and targeted masking to make pretrained language models location-aware.It avoids tokenizer modification, vocabulary expansion, architectural changes, and costly retraining.
- Conclusion: The approach improves alignment between embedding space and real-world geographic relationships while maintaining comparable performance on standard NLP benchmarks such as GLUE.The conclusion states that existing semantic and syntactic capabilities are not degraded.
- Conclusion: Around 15 minutes of additional training makes the adaptation computationally efficient for practical and scalable geo-spatial grounding.The stated efficiency is presented as supporting real-world applications.
7 Limitations
The study identifies unresolved limits in understanding the learned representations, the post-hoc adaptation setting, and coverage of generative language models.
- Limitations: The interaction between encoded location information and semantic or syntactic features requires deeper investigation.The limitation concerns understanding how geo-spatial information is represented within the embedding space.
- Limitations: Because the method adapts pretrained models post hoc, incorporating geo-spatial information during large-scale pretraining may produce stronger and more robust representations.The passage presents this as a future research direction rather than an established result.
- Limitations: The current work focuses on encoder-based models, leaving extension to generative language models as an open challenge.The authors identify generative-model extension as an important direction for future exploration.