Location reference identification from tweets during emergencies: A deep learning approach

Introduction

• Twitter’s user-location, place-name, and geo-coordinate fields have limitations and cannot be relied on completely.User locations may be arbitrary or outdated, place names may be absent or inaccurate, and geo-coordinates are infrequent and can be inauthentic.Source: §1 · p. 2
• Gazetteer lookup can miss uncovered regions and confuse location names with other meanings, while NER performs poorly on informal tweets with grammatical, abbreviation, and spelling errors.The paper favors word embeddings to preserve sentence structure and CNNs because tweets are short, while avoiding feature engineering required by conventional machine-learning models.Source: §1 · p. 3
• The study focuses on extracting location words, rather than all named entities, using a CNN instead of POS tagging.The tweet is treated as a sentence whose location-bearing words are highlighted.Source: §1 · pp. 3–4
• The supervised model assumes tweets have already been filtered for relevance to an event, then finds location-referring words in each tweet.Training uses tweets paired with their corresponding location words.Source: §1 · p. 4
• The objective is to determine whether a tweet contains a location name and, if so, highlight the location words.Source: §1 · p. 4

• Section 2 presents related literature, Section 3 the proposed framework, Section 4 the findings, Section 5 discussion and implications, and Section 6 the conclusion.Source: §1 · p. 4

2.1. Event detection

• Crisis research spans informative-message extraction, disaster-tweet classification, and early earthquake detection before official announcements (Imran et al., 2013; Imran et al., 2014b; Nguyen et al., 2017b).Source: §2.1 · p. 5

2.2. Location estimation

• Location estimation has used network, metadata, and tweet-text signals, while text-only methods address infrequent geo-tags and often outdated user-location fields (Do et al., 2017; Qian et al., 2017).Source: §2.2 · p. 5

2.2.1. Gazetteer Based Approach

• Gazetteer methods combined maps and volunteered data or n-gram matching; Middleton et al. (2014) reported street-level precision of 0.90 or above when preloading at-risk-area locations.Malmasi & Dras (2015) used noun-phrase extraction and GeoNames matching, achieving an F1-score of 0.792.Source: §2.2.1 · p. 6

2.2.2. Named Entity Recognition

• NER studies used clustering and grammar cues for event-location extraction, but Stanford NER missed local streets, buildings, abbreviations, misspellings, and lowercase names.Source: §2.2.2 · pp. 6–7

2.2.3. Efforts for Twitter Named Entity Recognition

• Twitter-adapted NER addressed domain mismatch: retrained Stanford NER reached an F1-score of 0.902, while conventional NER accuracy fell from 0.97 on news to 0.80 on tweets.Source: §2.2.3 · p. 7
• F1-scores were 0.85 for streets, 0.86 for buildings, 0.96 for location names, 0.88 for abbreviated place names, and 0.90 overall for location references.The work used lexico-semantic pattern recognition, NER, and a gazetteer to identify streets, buildings, and location names, respectively.Source: §2.2.3 · pp. 7–8
• F1-scores ranged from 0.90 to 0.97 for English and Italian tweets and were 0.66 for Turkish tweets using Middleton et al. (2018)'s location extraction approaches.Their approaches included OpenStreetMap entity matching, a language model using gazetteers and social-media tags, and third-party applications.Source: §2.2.3 · p. 8
• F1-score was 0.80 for Liu et al. (2013, 2011)'s framework, evaluated on 12,245 manually labeled tweets for person, product, location, and organization entities.The framework combined tweet normalization, K-Nearest Neighbors with linear Conditional Random Fields, and semi-supervised learning.Source: §2.2.3 · p. 8
• F1-score was 0.79 for Dutt et al. (2018)'s unsupervised extraction of location names from tweet text.After preprocessing, their system used POS tagging to identify proper nouns and a gazetteer-based approach to extract location names.Source: §2.2.3 · p. 8
• F1-scores were 65.89 for Limsopatham & Collier (2016)'s bidirectional LSTM and 40.78 for Däniken & Cieliebak (2017)'s transfer-learning approach.The LSTM learned tweet orthographic features using character-based and word-vector representations; the transfer-learning system used sentence-level features.Source: §2.2.3 · p. 8
• The proposed CNN eliminates feature extraction and POS tagging to identify location references in tweets.Earlier systems often required predefined features and location-specific gazetteers, making performance dependent on feature engineering.Source: §2.2.3 · p. 8

3. Methodology

• The overall architecture combines tweet embeddings, CNN feature extraction, and a fully connected output layer to predict location names in tweet text.The architecture is depicted in Figure 1.Source: §3 · p. 8, Figure 1 · p. 10

3.1. Data Collection, Preprocessing, and Labeling

• The researchers collected 103,384 earthquake tweets and annotated 5,107 tweets with 6,690 location references, retaining annotations agreed on by at least two of three students.Preprocessing removed non-English and duplicate tweets, usernames, URLs, and emoticons; hashtags were converted to words, text lowercased, stopwords retained, and other metadata discarded.Source: §3.1 · p. 9

3.2. Word embedding to represent tweets

• Tweets were represented as padded sequences of concatenated 100-dimensional GloVe embeddings, using vectors trained on 27 billion words of tweets.The vocabulary lookup matrix was initialized with the pretrained embeddings.Source: §3.2 · pp. 9, 11

3.3. Context dependent feature extraction

• The CNN extracts features from tweet word windows, applies ReLU and max-over-time pooling, then uses a dropout-regularized sigmoid layer to predict location-word probabilities.Convolutional features are flattened before the dense layer.Source: §3.3 · pp. 12–14

3.4. Representation of labels at the output layer

• Location references were encoded at the output layer as word-level binary labels, with location words assigned 1 and all other words assigned 0.The paper illustrates this scheme with tweets containing location words at specified word indices.Source: §3.4 · p. 14

3.5. Loss Function and Optimizer

• Binary cross-entropy with sigmoid outputs handles tweets containing multiple location labels, and Adam updates network weights by back-propagating the loss.The loss compares actual and predicted label values.Source: §3.5 · p. 14

3.6. Hyper-parameter setting

• Adam with binary cross-entropy produced the lowest training loss; the selected configuration used 128 filters per size, a pooling window of 5, and batch size 50.The model performed better with dropout 0.2 than 0.3 or 0.5; performance changed little after 100 epochs, so experiments used 100 epochs.Source: §3.6 · pp. 14–15

4. Result

• Ten-fold cross-validation trained on nine subsamples and validated on the tenth, repeating the process so each subsample served once for validation.Fold results were averaged to estimate overall performance.Source: §4 · p. 15
• Tweets were represented with 60 input neurons and 60 output neurons, corresponding to word positions and the presence or absence of location references.The authors report that most tweets contained at most 60 words.Source: §4 · p. 15

4.1. Evaluation metrics

• Evaluation used precision, recall, F1-score, Hamming loss, Jaccard similarity, and exact matching score; exact matching requires every location and non-location label to be correct.The section illustrates the metrics using a tweet containing four location words.Source: §4.1 · pp. 15–16

4.1.1. Filter size estimation

• 0.97 precision, 0.93 recall, 0.95 F1-score, 0.002 Hamming loss, 0.953 Jaccard similarity, and 0.924 exact matching score were obtained with combined 2-, 3-, and 4-gram filters.These filter sizes were fixed for subsequent experiments.Source: Table 4 · p. 17, §4.1.1 · p. 18
• Single 2-, 3-, 4-, and 5-gram filters yielded F1-scores of 0.52, 0.50, 0.52, and 0.53, respectively; two filters performed better than individual filters.The best filter combination used 2-, 3-, and 4-grams.Source: §4.1.1 · p. 18

4.1.2. Effect of varying the dense layers

• Two dense layers gave the best performance among the tested layer combinations, so the authors fixed the model at two dense layers for further experiments.The comparison used 2-CNN with 2-, 3-, and 4-gram filters.Source: Table 5 · p. 18, §4.1.2 · p. 18

4.1.3. Effect of varying the CNN layers

• The selected model uses three CNN layers, 2-, 3-, and 4-gram filters, two dense layers, and 0.2 dropout; three CNN layers gave the best tested result.Table 6 compares CNN-layer combinations using the three filter sizes.Source: Table 6 · p. 19, §4.1.3 · p. 19

• CNN-based location estimation spans streets to countries without POS tagging; proposed uses include early event localization and rescue operations.Source: §5 · pp. 19–20
• The best setup combined 2-, 3-, and 4-gram filters with 3-CNN, 2-Dense, and dropout.Source: §5 · p. 19
• 0.98 precision, 0.95 recall, 0.96 F1-score, 0.002 Hamming loss, 0.962 Jaccard similarity, and 0.929 exact matching score were achieved by the best model.Source: §5 · p. 19
• Multiword references require complete-span recovery: “Raoul Island, New Zealand” contains four location words whose individual pieces lack location meaning.Source: §5 · pp. 19–20
• Tweets sometimes contain five or more nonconsecutive location words, often separated by #, and adding 5-grams did not improve performance.Source: §5 · p. 20
• 92.9% exact matching required all predicted location- and non-location-referring words to match the actual labels.Source: §5 · p. 20
• Prior systems’ datasets were inaccessible, so the authors could not directly compare results and limited comparison to accuracy on those datasets.Source: §5 · p. 20
• The CNN model is independent of manual feature engineering, works with noisy tweets, and uses tweet text without metadata or user names to preserve privacy.Source: §5 · p. 20

6. Conclusion

• The CNN-based model targets location references in tweets, whose grammatical and spelling errors and non-standard abbreviations make extraction challenging.The evaluation used earthquake-related tweets.Source: §6 · p. 20
• F1-score 0.96 was the best result, achieved with 3-CNN and 2-Dense layers with dropout.The experiments used several convolution-layer and dense-layer configurations.Source: §6 · p. 21
• The model identifies location information at street, building, city, district, and country granularities.The conclusion describes its accuracy as impressive without reporting another metric here.Source: §6 · p. 21
• The authors identify road traffic management, sports events, and location-based services as possible applications, with domain-specific tweets for other domains.Trained models could also be integrated into mobile devices to find tweet locations on the fly.Source: §6 · p. 21
• The work considered only English tweets, while crisis users also post in regional languages; a deep neural network model could be created for multilingual tweets.Manual labeling requires substantial human effort, and semi-supervised approaches can reduce it to some extent.Source: §6 · p. 21