Source-linked AI summary
Geo-knowledge-guided GPT models improve the extraction of location descriptions from disaster-related social media messages
Yingjie Hu, Gengchen Mai, Chris Cundy, Kristy Choi, Ni Lao, Wei Liu, Gaurish Lakhanpal, Ryan Zhenqi Zhou, Kenneth Joseph
TL;DR
Disaster social media contains critical victim-location descriptions that typical NER tools may fail to extract completely, while training alternatives requires substantial labeled data. The paper combines geo-knowledge with GPT prompting, using 22 examples, and reports over 40% improvement over off-the-shelf NER approaches. It concludes that both geo-knowledge and GPT models are critical to the proposed approach.
Problem
Disaster-related location descriptions often extend beyond simple place names, challenging typical NER tools, while alternative machine-learning models require large, labor-intensive labeled datasets.
Method
The method creates GPT prompts from geo-knowledge about common location-description forms and uses them to recognize complete descriptions and their categories.
Results
Over 40% improvement over off-the-shelf NER approaches was achieved, and adding geo-knowledge produced an over 76% improvement compared with the same default GPT model.
Takeaways & Limitations
Geo-knowledge-guided GPT models provide an efficient approach for extracting location descriptions when large labeled datasets are costly to create, and both components are critical.
Takeaways & Limitations
The study focuses on Hurricane Harvey data from the Houston area, and further empirical research is needed to test applicability to other regions and disaster types.
Abstract
from arXiv · showhide
Social media messages posted by people during natural disasters often contain important location descriptions, such as the locations of victims. Recent research has shown that many of these location descriptions go beyond simple place names, such as city names and street names, and are difficult to extract using typical named entity recognition (NER) tools. While advanced machine learning models could be trained, they require large labeled training datasets that can be time-consuming and labor-intensive to create. In this work, we propose a method that fuses geo-knowledge of location descriptions and a Generative Pre-trained Transformer (GPT) model, such as ChatGPT and GPT-4. The result is a geo-knowledge-guided GPT model that can accurately extract location descriptions from disaster-related social media messages. Also, only 22 training examples encoding geo-knowledge are used in our method. We conduct experiments to compare this method with nine alternative approaches on a dataset of tweets from Hurricane Harvey. Our method demonstrates an over 40% improvement over typically used NER approaches. The experiment results also show that geo-knowledge is indispensable for guiding the behavior of GPT models. The extracted location descriptions can help disaster responders reach victims more quickly and may even save lives.
1. Introduction
Disaster-related social media contains urgent location information, but typical NER methods struggle with complete, multi-entity descriptions and detailed location categories. The paper proposes prompts that combine geo-knowledge with GPT models to address these limitations using few training examples.
- 89 natural disasters in the United States caused over 4,500 deaths and more than $780 billion in damages and losses between 2017 and 2021.
- Social media location descriptions can help responders reach victims more quickly while reducing the time and labor required for manual screening.
- Recognizing location descriptions is the prerequisite for geo-locating them into geographic coordinates and spatial representations.
- Typical NER approaches may split multi-entity location descriptions instead of recognizing them as complete descriptions.
- Previous studies generally do not classify recognized descriptions into categories such as door-number addresses, road intersections, and road segments.
- The proposed geo-knowledge-guided prompt uses common forms of location descriptions and feeds the resulting prompt to GPT through question answering.The approach is designed to recognize full descriptions and identify their categories using only a small number of training examples.
2. Related work
Related work uses social media for disaster response and extracts locations from message content, but location-description recognition differs from conventional geoparsing. Existing work often emphasizes geotagged locations, NER-based extraction, or toponym resolution.
- Social media research supports disaster response through near-real-time information, including location extraction, event analysis, and situational awareness.
- Previous studies often focused on geotagged locations rather than locations described in tweet content.
- Researchers have used pre-trained NER tools such as Stanford NER and SpaCy NER to extract locations from tweet content.
- Geospatial research on prepositions can support the later geo-locating step for recognized location descriptions.
- Disaster-related location-description extraction is related to but different from geoparsing, which typically recognizes and resolves toponyms in two steps.Geoparsing research often emphasizes toponym resolution and place-name ambiguity, whereas this paper focuses on recognition of descriptions in disaster messages.
3. Method
The method combines geo-knowledge about common location-description forms with GPT models through prompts to recognize complete descriptions and their categories. Its knowledge base uses 11 categories and examples, while the prompt provides 22 tweet examples for newer GPT models.
- Method overview: The proposed method fuses geo-knowledge, a GPT model, and a question-answering process to recognize complete location descriptions and their categories.Geo-knowledge is encoded into prompts, and GPT-3, ChatGPT, and GPT-4 are tested.
- Geo-knowledge about location descriptions: Geo-knowledge was extended from a Hurricane Harvey tweet study that manually annotated location descriptions in a sampled dataset.The prior study sampled 1,000 tweets from 15,834 candidate tweets selected from more than 7 million disaster-related tweets.
- Geo-knowledge about location descriptions: The knowledge scheme contains 11 categories, including administrative units, multiple areas, and road segments, represented through typical forms and examples.Road segments were added because they may be better represented geometrically as lines, unlike intersections and highway exits, which are represented as points.
- Geo-knowledge about location descriptions: The categorization remains revisable because uncommon location descriptions may not fit the current categories, while the prompting framework can accommodate updated examples.The authors describe the current categorization as one version of their geo-knowledge.
- GPT models: The study tests whether systematic geo-knowledge can guide GPT behavior across GPT-2, GPT-3, ChatGPT, and GPT-4 models.The authors hypothesize that prompts based on knowledge of disaster-related location-description forms are more effective for the target task.
- Fusing geo-knowledge and GPT: The prompt uses question-answering statements based on category knowledge and examples, with 22 tweet examples spanning 11 categories for the full prompt.A prompt snippet appears in Table 2, and the complete prompt is provided in Supplementary Table S1.
4. Evaluation Experiments
The evaluation uses annotated Hurricane Harvey tweets to compare geo-knowledge-guided GPT implementations with GPT, BERT, and NER alternatives. It measures recognition using precision, recall, and F-score with full-span matching.
- Experiment design: The experiments compare implementations using different GPT models and nine alternatives, including NER, fine-tuned BERT, and default GPT models without geo-knowledge.The comparison is designed to assess both the proposed method and the role of geo-knowledge in GPT guidance.
- Experiment dataset: The test set contains 978 Hurricane Harvey tweets after excluding the 22 tweets used to create the prompt from the 1,000 annotated tweets.The dataset includes complete location descriptions and location categories, and annotations use the IOB tagging scheme.
- Experiment models: Geo-GPT-4, Geo-ChatGPT, Geo-GPT-3, and Geo-GPT-2 use prompts encoding geo-knowledge, with GPT-2 receiving 11 examples because of its 1024-token prompt limit.The newer GPT implementations use the same prompt based on 22 tweet examples.
- Experiment models: Default GPT-4, ChatGPT, GPT-3, and GPT-2 baselines receive the same location-recognition question without additional geo-knowledge and cannot identify location categories.They can still detect location descriptions based on text encountered during pre-training.
- Evaluation metrics: Precision, recall, and F-score evaluate model performance, with correctly recognized descriptions determined by full-span matching.Precision measures correctness among recognized descriptions, recall measures coverage of annotated descriptions, and F-score is their harmonic mean.
5. Results
The experiments compare NER, GPT, Geo-GPT, and fine-tuned BERT models for extracting complete location descriptions, with and without category recognition. Geo-knowledge-guided GPT models outperform corresponding baselines, while strict full-span evaluation reveals meaningful boundary-matching errors.
- 5.1 Ability to recognize complete location descriptions regardless of categories: Four NER models achieve precisions of 0.352-0.643, recalls of 0.224-0.440, and F-scores of 0.332-0.495, but miss multi-entity descriptions.They recognize simple place names while failing on door number addresses, road segments, and road intersections.
- 5.1 Ability to recognize complete location descriptions regardless of categories: Geo-GPT models substantially improve over corresponding default GPT models in all metrics across GPT-2 through GPT-4.Default GPT models recognize only about 25%-40% of descriptions, with GPT-4 reaching a highest F-score of 0.394; Geo-knowledge supplies common forms and categories of location descriptions.
- 5.1 Ability to recognize complete location descriptions regardless of categories: 0.695 is the highest F-score, achieved by Geo-GPT-4, while Geo-GPT-3 achieves 0.693 and both approach the 0.70 disaster-response acceptability threshold.These scores represent an over 40% improvement over off-the-shelf NER models and use only a small number of training examples encoding geo-knowledge.
- 5.2 Ability to recognize both complete location descriptions and location categories: Geo-GPT-3, Geo-ChatGPT, and Geo-GPT-4 outperform Geo-GPT-2 and fine-tuned BERT across all eleven location-description categories and overall.The second experiment counts a recognition as correct only when both the complete description and its category are correct.
- 5.2 Ability to recognize both complete location descriptions and location categories: Geo-GPT-3 and Geo-GPT-4 achieve precisions, recalls, and F-scores of about 0.75 for door number addresses.They sometimes extract whole addresses including prepositions, supporting subsequent analysis and geo-location.
- 5.2 Ability to recognize both complete location descriptions and location categories: Geo-GPT-4 errors commonly involve road segments being classified as street names or intersections, while strict full-span matching penalizes reasonable wording differences.The confusion matrix uses rows for ground-truth categories, columns for model predictions, and diagonal cells for correct recognitions.
6. Discussion
The discussion presents geo-knowledge-guided GPT models as effective for extracting and categorizing disaster-related location descriptions, while emphasizing transferability, operational implications, ethical concerns, and remaining limitations.
- 6.2 Implications for research in other geographic regions and data from other platforms: The method can be adapted to other regions by replacing U.S. geo-knowledge with examples representing local location descriptions.The authors state that this adaptation may require identifying and organizing local descriptions when regional geo-knowledge is unavailable.
- 6.2 Implications for research in other geographic regions and data from other platforms: Because it uses textual content rather than Twitter-specific features, the method may also analyze messages from other platforms.The authors specifically identify platform-independent textual processing as the basis for this possible extension.
- 6.3 A potential paradigm shift for using AI models for disaster response?: Prompt-guided online GPT models could reduce organizations’ needs for labeled data, local computing environments, technical expertise, and development time.The discussion also frames this approach as enabling disaster experts and model developers to contribute complementary expertise.
- 6.3 A potential paradigm shift for using AI models for disaster response?: Submitting disaster victims’ messages to online GPT models raises concerns about increased exposure, data ownership, evidence access, and stronger protections for emergency data.The authors call for additional policies and technical measures to safeguard content submitted to AI models.
- 6.4 Limitations: The method remains limited by confusion among some location categories, a Houston Hurricane Harvey dataset, and the need for further methodological and geographic validation.The authors specifically identify confusion between road segments, street names, and intersections, alongside limited overall performance.
7. Conclusions
The conclusion reports a geo-knowledge-and-GPT method for recognizing disaster-related location descriptions and categories, with systematic experiments showing improvements over NER and default GPT approaches.
- 7. Conclusions: The study proposes fusing geo-knowledge and GPT models to recognize location descriptions and their categories in disaster-related social media messages.The proposed approach is evaluated through systematic experiments against alternative methods.
- 7. Conclusions: Over 40% improvement was achieved over off-the-shelf NER approaches for recognizing location descriptions, using only a small number of geo-knowledge training examples.The study also reports over 76% improvement after adding geo-knowledge to the same default GPT model.
- 7. Conclusions: Geo-GPT-4 achieved an F-score of 0.695 for recognizing location descriptions regardless of categories and an overall F-score of 0.644 for recognizing descriptions and categories.The highest reported performance came from Geo-GPT-4 among the evaluated approaches.
- 7. Conclusions: Geo-GPT-4 achieved an F-score of 0.755 for the door number address category.The authors describe this category as providing highly detailed information for locating victims.
Funding
The work was supported by the U.S. National Science Foundation through Grant No. BCS-2117771.
- Funding: The U.S. National Science Foundation supported this work through Grant No. BCS-2117771.The grant concerns geospatial artificial intelligence approaches for understanding location descriptions in natural disasters and their spatial biases.
Notes on contributors
The contributors span geography, computer science, computational social science, machine learning, GeoAI, and data analysis, with roles covering conceptualization, methodology, data work, interpretation, and writing.
- Notes on contributors: Yingjie Hu contributed conceptualization, data collection and curation, methodology, analysis, interpretation, visualization, and writing.He is an Associate Professor of Geography at the University at Buffalo whose research includes GIScience, GeoAI, and disaster resilience.
- Notes on contributors: Gengchen Mai contributed conceptualization, methodology, data analysis, interpretation, visualization, and writing.His research includes spatially explicit AI, geographic knowledge graphs, geographic question answering, and geospatial foundation models.
- Notes on contributors: Chris Cundy, Kristy Choi, and Ni Lao contributed methodology, result interpretation and discussion, and writing.Their research backgrounds include generative models and AI safety, limited-labeled-supervision machine learning, and knowledge graphs and natural language understanding.
- Notes on contributors: Wei Liu, Gaurish Lakhanpal, and Ryan Zhenqi Zhou contributed data collection and curation or data analysis, with Lakhanpal also contributing visualization.Their research interests include GeoAI, urban analytics, machine learning, disaster resilience, public health, and human mobility.
- Notes on contributors: Kenneth Joseph contributed result interpretation, discussion, and writing as a computational social scientist.He is an Assistant Professor in Computer Science and Engineering at the University at Buffalo.
- Notes on contributors: The paper states that its supporting data and code are available on figshare.The availability statement provides the DOI https://doi.org/10.6084/m9.figshare.22659337.
Supplementary Materials
The supplementary materials provide complete prompts built from geo-knowledge of common location-description forms, including a second prompt using a different set of 22 tweet examples. The examples span addresses, roads, intersections, exits, landmarks, waterways, cities, and broader areas.
- Table S1 presents a complete prompt based on geo-knowledge of common forms of location descriptions.
- The prompt examples include simple locations such as cities, states, roads, highways, waterways, parks, schools, hospitals, and civic centers.
- Several examples represent relational or compound descriptions, including intersections, roads relative to waterways, highway exits, and road segments.
- Other examples cover detailed addresses and named facilities, including apartment addresses, shelters, churches, schools, and parking lots.
- Table S2 supplies a different set of 22 tweet examples while retaining the same geo-knowledge.
- The examples also include broader areas and geographic combinations, such as neighborhood areas, city pairs, reservoirs, and multiple landmarks or roads in one message.