PaperlayerView exact revision on arXiv

Source-linked AI summary

HUMAID-NER: A Disaster Tweet Dataset for Joint Named Entity Recognition and Event Classification via Uncertainty-Weighted Multitask Learning

Aijaz Ali, Nazish Basir, Sarfaraz Nawaz, Danish Nazir Arain, Haris Ali

arXiv:2609.16964v1cs.CLcs.LG

TL;DR

Disaster tweet resources provide document-level categories but lack span-level entity annotations for extracting operational details. The paper introduces HUMAID-NER and a shared-encoder multitask framework for disaster-specific NER and humanitarian event classification. On validation, the model achieves both NER span micro-F1 of 0.841 and classification macro-F1 of 0.761.

  • Problem

    Existing disaster tweet resources mainly provide category labels, leaving span-level entity annotations unavailable for extracting details such as locations, casualty counts, and resource needs.

    Source: §I · p. 2

  • Method

    The paper builds HUMAID-NER with a three-stage hybrid annotation pipeline and trains joint NER and classification heads on a shared RoBERTa-large encoder with uncertainty weighting.

    Source: §I · p. 2, A. Dataset: HUMAID-NER · p. 3

  • Results

    NER span micro-F1 of 0.841 and classification macro-F1 of 0.761 are achieved simultaneously on the HUMAID-NER validation set.

    Source: §V · p. 7

  • Takeaways & Limitations

    A real-time dashboard demonstrates simultaneous entity extraction and event classification from live tweet input.

    Source: §I · p. 2

    The annotations lack human inter-annotator validation, and evaluation is limited to the validation split, English tweets from 2016–2019, and single-run ablation estimates.

    Source: §V · p. 8

Abstract

from arXiv · show

Rapid extraction of structured information from social media is important for humanitarian response, yet existing disaster tweet resources mainly provide document-level category labels without span-level entity annotations. We introduce HUMAID-NER, the first named entity recognition dataset built on the HumAID benchmark, containing 60,000 English disaster tweets annotated in BIO format across ten operationally motivated entity types and yielding approximately 175,000 labelled entity spans. Annotations are generated through a reproducible three-stage hybrid pipeline combining a spaCy transformer model, disaster-domain EntityRuler patterns, and structured regular expressions with priority-based overlap resolution. We also propose a joint multitask learning framework that performs disaster-specific named entity recognition and humanitarian event classification using a shared RoBERTa-large encoder. To reduce task conflict during joint training, the model uses homoscedastic uncertainty weighting with learnable task parameters and a two-stage training schedule that freezes the lower 18 of 24 encoder layers in the second stage. On the HUMAID-NER validation set, the proposed system achieves NER span micro-F1 of 0.841 and classification macro-F1 of 0.761 simultaneously. A real-time web dashboard demonstrates end-to-end deployment. The dataset, models, and pipeline code are released to support reproducibility and future crisis informatics research.

I. INTRODUCTION

Disaster tweet classification identifies message categories but does not extract the locations, casualty counts, or resource needs that response coordinators seek. The paper addresses the lack of disaster-specific NER annotations and joint NER-classification models with a new dataset and multitask framework.

I. INTRODUCTION

  • The paper identifies a gap in disaster NLP: existing resources lack disaster-specific span annotations, and prior work has not jointly modeled disaster NER and event classification.Generic social-media NER taxonomies omit operational types such as CASUALTY, DISPLACED, RESOURCE, RESCUE, and REQUEST.Source: §I · p. 2
  • Fixed-weight joint training produced a structural task conflict: classification macro-F1 peaked at epoch three and fell 1.4 points by epoch fifteen.The authors attribute the imbalance to dense token-level NER supervision versus one sentence-level classification signal per example.Source: §I · p. 2
  • HUMAID-NER extends HumAID with 60,000 tweets annotated in BIO format across ten disaster-specific entity types.The proposed joint framework combines RoBERTa-large, uncertainty weighting, and two-stage layer-freezing training.Source: §I · p. 2
  • The paper also deploys the joint model in a real-time dashboard for simultaneous entity extraction and event classification from live tweets.The dashboard is presented as an end-to-end demonstration beyond academic evaluation.Source: §I · p. 2

II. RELATED WORK

Prior disaster-tweet datasets and classifiers largely provide document-level categories, while social-media NER work motivates domain-specific entity types. Related multitask research informs the paper’s shared-encoder and training design.

A. Disaster Tweet Analysis

  • Disaster resources including HumAID, CrisisMMD, and CrisisNLP provide categorical labels but no span-level entity annotations.HumAID covers 19 disasters from 2016–2019 with roughly 77,000 tweets across ten humanitarian categories.Source: A. Disaster Tweet Analysis · p. 2

B. Named Entity Recognition on Social Media

  • Social-media NER research documents transfer problems from newswire and motivates entity categories grounded in humanitarian-response vocabulary.HUMAID-NER draws on that operational vocabulary rather than inheriting general-purpose newswire categories.Source: B. Named Entity Recognition on Social… · p. 2

C. Multitask Learning for NLP

  • Shared-task learning is motivated by vocabulary overlap in disaster tweets and prior findings that lower encoder layers learn universal features while upper layers encode task-specific patterns.MT-DNN’s layer-function insight directly motivates the paper’s two-stage training procedure.Source: C. Multitask Learning for NLP · p. 3

D. Task Weighting in Multitask Learning

The paper motivates learnable uncertainty weighting as a low-overhead way to balance task losses. It extends a balanced HumAID subset with disaster-specific BIO annotations and a hybrid pipeline for assigning them.

D. Task Weighting in Multitask Learning

  • Uncertainty weighting uses learnable log-variance parameters to scale each task’s loss, adding two scalar parameters with negligible overhead.The paper contrasts this with PCGrad’s roughly doubled memory cost and GradNorm’s increased per-step compute; a direct disaster-domain comparison remains open.Source: D. Task Weighting in Multitask Learning · p. 3

A. Dataset: HUMAID-NER

  • HUMAID-NER annotates a balanced 60,000-tweet HumAID subset with BIO spans across ten disaster-specific entity types and stratifies train, validation, and test splits 72/14/14.HumAID contains 77,637 English tweets from 19 major disasters and has category labels but no span-level annotations.Source: A. Dataset: HUMAID-NER · p. 3
  • A three-stage hybrid pipeline combines spaCy predictions, disaster-domain EntityRuler patterns, and regular expressions, then resolves overlaps and converts annotations to BIO format.Longer competing spans are retained; equal-length conflicts prioritize neural predictions over regex matches.Source: A. Dataset: HUMAID-NER · p. 3, Figure 1 · p. 3

B. Shared Encoder

The model uses a shared RoBERTa-large encoder with task-specific heads for token-level NER and humanitarian-category classification. Its classification head combines [CLS] and mean-pooled token representations.

B. Shared Encoder

  • RoBERTa-large provides a 24-layer, 1,024-dimensional shared encoder with approximately 355 million parameters.DeBERTa-v3 was evaluated but rejected because incompatible XLA gather operations on TPU v3-8 prevented stable training.Source: B. Shared Encoder · pp. 3–4

C. Task-Specific Heads

  • The NER head classifies tokens into 21 BIO classes, while the classification head concatenates [CLS] and mean-pooled representations to predict ten humanitarian categories.The combined task loss is weighted by learnable uncertainty parameters.Source: Figure 2 · p. 4, C. Task-Specific Heads · p. 4

D. Uncertainty-Weighted Multitask Objective

The multitask objective addresses the imbalance between dense token-level NER supervision and sentence-level classification by weighting task losses according to learnable uncertainty. Both task losses decrease monotonically as the uncertainty parameters adapt.

D. Uncertainty-Weighted Multitask Objective

  • The NER loss averages cross-entropy over non-padding token positions; N counts these positions, while yic and pic denote the ground-truth indicator and predicted probability.Source: D. Uncertainty-Weighted Multitask… · p. 4
  • Homoscedastic uncertainty weighting replaces fixed task weights with learnable log-variance parameters that scale task losses inversely with uncertainty.The NER loss averages standard cross-entropy over non-padding token positions.Source: D. Uncertainty-Weighted Multitask… · p. 4
  • Both uncertainty parameters rise from 0 to approximately 1.26 by epoch 15, while NER and classification losses decrease monotonically.The authors report that neither task is neglected as the uncertainty parameters adapt.Source: D. Uncertainty-Weighted Multitask… · p. 4

E. Two-Stage Training Procedure

Training proceeds in two stages: all encoder layers first learn jointly, then the lower 18 layers are frozen while the upper six layers and task heads are fine-tuned. This schedule reflects the distinction between shared lower-layer features and task-specific upper-layer patterns.

E. Two-Stage Training Procedure

  • The schedule treats lower encoder layers as learning universal features and upper layers as encoding task-specific patterns.Source: E. Two-Stage Training Procedure · p. 5
  • Stage 1 trains all 24 encoder layers, both task heads, and the uncertainty parameters jointly during epochs 1–7.The boundary was set empirically after the validation combined score plateaued between epochs 6 and 8.Source: E. Two-Stage Training Procedure · p. 5
  • Stage 2 freezes layers 1–18 during epochs 8–15, leaving layers 19–24, both task heads, and uncertainty scalars trainable.Active parameters fall from 355M to roughly 47M.Source: E. Two-Stage Training Procedure · p. 5

F. Optimisation and Evaluation

The paper evaluates span-level NER and category classification with separate F1 measures, selecting checkpoints using a balanced combined score. Results use a held-out validation split, while configuration selection prioritizes classification macro-F1 for deployment routing.

F. Optimisation and Evaluation

  • NER is measured by entity-level span micro-F1 requiring exact boundary and type matches, while classification uses macro-F1 across ten humanitarian categories.Source: F. Optimisation and Evaluation · p. 5
  • CLS macro-F1 is the primary criterion for selecting configurations, with the combined score breaking ties because humanitarian response routing depends on event category labels.Source: F. Optimisation and Evaluation · p. 5

IV. RESULTS AND DEMONSTRATION

  • 8,400 held-out validation tweets supply all reported results, using the best checkpoint selected by Eq. (5).Source: §IV · p. 5

A. Ablation Study

The ablation favors Row D under the classification-first deployment policy, while showing trade-offs across encoder choice, uncertainty weighting, and training schedule. Under fixed weights, CLS validation performance declines even as NER improves and training losses fall.

A. Ablation Study

  • The controlled ablation adds one component per row: RoBERTa-large, uncertainty weighting, then two-stage freezing, following the BERT-large fixed-weight baseline.Fig. 4 displays NER span micro-F1, CLS macro-F1, and combined score for Rows A–D.Source: A. Ablation Study · p. 5, Figure 4 · p. 6

B. Component Analysis

  • Switching from BERT-large to RoBERTa-large raises CLS by 1.3 points (0.736→0.749) while reducing NER by 1.0 point (0.873→0.863).The passage attributes the NER reduction to BERT-large’s NER head saturating at epoch 12 under extended training.Source: B. Component Analysis · p. 5
  • Under fixed weights, CLS fell from 0.748 to 0.736 as NER rose from 0.711 to 0.873; Kendall weighting’s primary value is preventing this degradation.Direct gradient diagnostics are left for future work.Source: B. Component Analysis · p. 5, Figure 5 · p. 6
  • Row D gains +1.6 CLS points (0.745→0.761) at a 2.5-point NER cost and is preferred under the classification-first deployment policy.It achieves the highest CLS macro-F1 across Rows A–D.Source: B. Component Analysis · p. 5, Figure 4 · p. 6

C. Task-Conflict Analysis

  • Row A’s CLS validation F1 fell 1.4 points while both training losses declined, a pattern consistent with negative transfer; direct gradient diagnostics remain future work.The decline unfolded progressively, while NER showed no corresponding decline.Source: C. Task-Conflict Analysis · p. 5, Figure 5 · p. 6

D. Convergence and Checkpoint Selection

The combined validation score follows a non-monotonic trajectory under single-stage training, motivating exhaustive checkpoint sweeps. Training losses can continue falling while validation CLS performance degrades.

D. Convergence and Checkpoint Selection

  • Combined validation score plateaued at 0.791–0.799 during epochs 5–8, then peaked at 0.8045 at epoch 12, motivating exhaustive sweeps rather than early stopping.The same sweep policy was applied across all rows.Source: D. Convergence and Checkpoint Selection · p. 6
  • Both task losses decrease monotonically even as validation CLS F1 degrades after epoch 3, so training loss alone does not establish generalization.The curves are for Row A.Source: Figure 6 · p. 6

E. Final Validation Results

The proposed joint system reports concurrent disaster-domain entity extraction and event classification, with a real-time dashboard and a reproducible dataset pipeline. Its validation results are qualified by the smaller training subset and other stated evaluation limitations.

E. Final Validation Results

  • NER span micro-F1 of 0.841 and CLS macro-F1 of 0.761 are achieved concurrently, with classification meeting or exceeding dedicated single-task HumAID classifiers.The comparison is qualified because the joint model used a balanced 60,000-tweet subset, while the cited baselines used 77,637 tweets.Source: E. Final Validation Results · pp. 6–7, Table IV · p. 7

F. Real-Time Deployment Dashboard

V. CONCLUSION

  • HUMAID-NER is the first disaster-tweet NER dataset, and its three-stage hybrid pipeline is reproducible and scalable to all 19 HumAID events.The conclusion describes the annotations as BIO-format and spanning ten operationally motivated entity types.Source: §V · p. 7
  • Under fixed weights, CLS fell 1.4 points across fifteen epochs; two-stage freezing yielded the largest single-component CLS gain (+1.6 points).The proposed explanation for fixed-weight task conflict is the 2,688-to-1 supervision-count ratio, but direct gradient diagnostics were not collected; uncertainty weighting’s main value was training stability.Source: §V · p. 7
  • Reported limitations include no human inter-annotator validation, English-only coverage from 2016–2019, validation-split-only reporting, and single-run ablation estimates.The paper states that multi-seed variance analysis was precluded by TPU compute budget.Source: E. Final Validation Results · p. 7, §V · p. 8

ETHICS STATEMENT

The study uses publicly available tweets under their original terms and reports no new data collection or human-subject involvement. It advises independent validation before decision-critical operational use.

ETHICS STATEMENT

  • The dataset uses publicly available tweets released under their original terms of use.No new data collection or human-subject involvement took place.Source: ETHICS STATEMENT · p. 8
  • The auto-labelling pipeline neither deanonymises users nor infers personal attributes.Source: ETHICS STATEMENT · p. 8
  • Operational deployment in decision-critical workflows should follow independent validation on data from the relevant disaster types and languages.Source: ETHICS STATEMENT · p. 8

Related papers

Loading 2609.16964v1…