Source-linked AI summary
CrisiLangs: A Multilingual Resource for Crisis Warning Generation
Giacomo Gonella, Stefano Menini, Marco Guerini
TL;DR
Crisis warning generation resources have covered English, despite the need to communicate promptly across languages during emergencies. CrisiLangs adds four languages and benchmarks joint multilingual fine-tuning against a zero-shot baseline. Fine-tuning improves automatic metrics across languages and configurations, while human preferences favor lower decoding temperatures and vary on guidelines.
Problem
Existing crisis warning generation resources cover English, while multilingual crisis communication is needed and translation quality can be lower for low-resource languages and English-to-other-language translation.
Source: §1 · p. 1Method
CrisiLangs extends CrisiText with native-speaker translations of warning messages in Italian, Bulgarian, German, and Greek, then benchmarks joint five-language fine-tuning with and without guidelines against zero-shot generation.
Source: §1 · p. 1, §3 · pp. 3–4Results
Takeaways & Limitations
Native-speaker assessment remains important because annotators found Bulgarian +guidelines outputs unacceptable in 13 of 50 items despite automatic scores close to the plain configuration.
Source: §7 · p. 9The study does not evaluate feeding English events directly to generate target-language warnings, a setting that would remove automatic translation from the input pipeline.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Communicating with the public during an emergency entails issuing warning messages promptly. In Europe, it often requires producing them in more than one language. Recent work has shown that language models can be trained to generate warning messages as a crisis unfolds, but the available data covers only English. To fill this gap, we present CrisiLangs, a multilingual extension of CrisiText (an English dataset of crisis events paired with warning messages written according to expert communication guidelines) to Italian, Bulgarian, German, and Greek. Expert native speakers translated the warning messages of 1,000 events drawn from 411 scenarios, for a total of 5,000 event-message pairs across five languages. We test the resource to fine-tune a multilingual model on all five languages jointly, with and without the emergency guidelines in the prompt, and compare it with a zero-shot baseline. The fine-tuned models outperform the baseline on ROUGE-L, BLEU and BERTScore in every language and configuration. In a human preference study, low decoding temperatures receive the most preferences in all five languages and in both configurations, while the effect of the guidelines varies by language.
1 Introduction
Crisis warnings must reach people quickly in linguistically diverse settings, but existing generation resources cover only English. CrisiLangs extends that resource to four additional languages and is used to benchmark multilingual fine-tuning.
1 Introduction
- Emergency authorities must repeatedly tell the public what is happening and what to do, while producing multilingual alerts quickly adds to operators’ workload.The paper links this challenge to time pressure and the volume of information handled during emergencies.Source: §1 · p. 1
- Existing crisis warning generation resources such as CrisiText cover only English, while translation-based approaches face quality concerns, especially for low-resource languages and English-to-other-language translation.The paper therefore identifies a need for training resources in languages beyond English.Source: §1 · p. 1
- CrisiLangs extends CrisiText to Italian, Bulgarian, German, and Greek with 1,000 events from 411 scenarios and 5,000 event-message pairs across five languages.The added languages were selected to cover different writing systems and alphabets.Source: §1 · pp. 1–2
- The authors jointly fine-tune a multilingual model on all five languages, with and without emergency guidelines in the prompt, and evaluate it against a zero-shot baseline.Evaluation uses automatic metrics at five decoding temperatures and native-speaker assessment.Source: §1 · p. 2
2 Related Work
Crisis NLP has largely focused on analyzing user-generated text, with generation work addressing other outputs or English-only warning messages. Translation research tackles multilingual crisis communication, but it does not provide the same direct warning-generation setting.
2 Related Work
- Many crisis NLP resources support classification and information extraction from user-created content, and comparatively few address languages beyond English.Examples include multilingual humanitarian documents, Arabic crisis tweets, and cross-lingual classification or help-request detection.Source: §2 · p. 2
- Crisis generation work includes daily summaries and emergency-call support, but those outputs derive from existing reports or calls rather than directly addressing the public with warnings.The cited work also includes replies to social-media posts and personalized evacuation guidance.Source: §2 · pp. 2–3
- Crisis translation research covers planning and machine translation, but reported translation quality degrades particularly for low-resource languages and English-to-other-language translation.The cited studies evaluate crisis-domain translation systems and propose simplified English when full multilingual coverage is infeasible.Source: §2 · p. 3
- CrisiText pairs chronological crisis events with guideline-based English warnings for generation, while related warning-generation work also remains English-only.Other cited settings generate replies to individual social-media posts or narrowcast evacuation guidance.Source: §2 · p. 3
3 Multilingual Dataset
CrisiLangs extends CrisiText using a manually selected, expert-reviewed sample and native-speaker translations of the warning messages. Event descriptions and guidelines are machine-translated, while the resulting resource covers five languages and 411 scenarios.
Multilingual Translation
- Four native speakers, one per target language, translated and adapted the warning messages, while event descriptions and FEMA guidelines were machine-translated.Translators could consult the English source, and “9-1-1” was replaced with “112” for the European audience.Source: §3 · pp. 3–4
Source Data
- CrisiText provides more than 100,000 event-message pairs, with crisis categories and chronological event chains; its messages were synthetically generated using expert communication guidance.Scenarios remain within one original train, development, or test split.Source: §3 · p. 3
Selected Data
- The multilingual extension uses 1,000 expert-reviewed pairs sampled from the original splits in an 80/10/10 proportion, balanced between 500 human-caused and 500 natural events.Events may have multiple labels, but the category distribution reports only each event’s main category.Source: §3 · p. 3, Table 1 · p. 4
Multilingual Translation
- Human translation was reserved for warning messages because they are the model’s output and the object of human evaluation; input-side translation was treated as less critical.The input-side approach also reflects a deployment setting in which input is processed automatically.Source: §3 · p. 4
- The resource contains 5,000 event-message pairs across five languages and 411 scenarios, with machine-translated events alongside human-translated messages in the four added languages.Parallel versions differ slightly in mean message length by language.Source: Figure 2 · p. 4, §3 · p. 5
4 Experimental Design
The experiments jointly fine-tune one multilingual model in plain and guideline-prompt configurations, then compare generated warnings with a zero-shot baseline. Automatic metrics and a native-speaker preference study assess output quality across languages and decoding settings.
Model and Training Data Construction
- The study fine-tunes Llama-3.1-8B-Instruct with LoRA jointly on all five languages, using event sequences as context and generating a warning for the final event.The prompt keeps its instructions in English and presents event descriptions in the target language.Source: §4 · p. 5
Training Configurations
- The plain prompt contains event descriptions, while the +guidelines prompt also includes translated FEMA guidance for the event’s crisis categories.The comparison tests whether generation-time guidance is needed or whether fine-tuning lets the model infer it.Source: §4 · p. 5, Figure 3 · p. 5
Generation and Automatic Metrics
- The authors generate warnings for 100 test events in five languages at five temperatures and compare both fine-tuned configurations with a zero-shot baseline using ROUGE-L, BLEU, and BERTScore.The automatic scores also determine which generations enter the human study.Source: §4 · p. 5, Table 3 · p. 6
Human Preference Study
- Native speakers evaluate three selected temperature outputs for 50 test event-message pairs per language and configuration, judging grammaticality and task compliance.Candidates are anonymized, and annotators also report their overall configuration preference.Source: §4 · pp. 5–6
5 Results
The results show that fine-tuning improves automatic scores over the zero-shot baseline, while the human study distinguishes temperature preferences and language-specific differences between prompt configurations.
5.1 Automatic Evaluation
- Fine-tuned models score higher than the baseline on ROUGE-L, BLEU, and BERTScore in every language, temperature, and prompt configuration.At T=0.2, fine-tuning raises BERTScore from .892 to .916; BLEU’s n-gram precision term rises from .114 to .177, and generation length matches the references.Source: §5.1 · p. 6
- No training configuration is consistently better across languages by automatic metrics, and pooled scores are virtually identical.In the 0.6–0.2 range, metric preferences vary by language; BERTScore differences are at most .003 per language.Source: §5.1 · p. 7
5.2 Human Preference Study
- Temperature 0.2 receives the most human preferences in all five languages and both training configurations, though the margin is small for Greek with guidelines.The study compared generations at temperatures 0.6, 0.4, and 0.2 across 50 test events.Source: §5.2 · p. 7
- Annotator preferences varied by language, while comments identified wording and action appropriateness as well as grammatical errors and unfinished text.Four annotators preferred the plain model overall; the Greek annotator preferred the guidelines model, and comments about defects concentrated on that model in Bulgarian and German.Source: §5.2 · pp. 7–8
6 Discussion
Fine-tuning CrisiLangs produces one model for all five languages that exceeds the baseline in automatic evaluation, but the human study reveals uneven quality and a Bulgarian reliability concern.
6 Discussion
- For Bulgarian warning generation, annotators could always make a choice with the plain model, while 13 items were left unannotated with the +guidelines model because no candidate was acceptable.The authors describe the +guidelines output as less reliable, mostly because of LLM limitations.Source: §6 · p. 8
- Among the three temperatures compared, 0.2 receives the most preferences in every language and both configurations, making lower temperatures the safer practical choice.Annotators judged grammaticality and task compliance; the preference pattern held with and without guidelines in the prompt.Source: §6 · p. 8
- Guidelines in the prompt are not consistently beneficial: four annotators preferred the plain model, while the Greek annotator favored guidelines.Automatic metrics also show no consistent advantage for either configuration.Source: §6 · p. 8
- Automatic metrics did not anticipate Bulgarian acceptability failures: scores were close across configurations despite 13 of 50 guideline-model items being rejected.The authors conclude that human validation remains necessary when extending crisis warning generation to new languages.Source: §6 · pp. 8–9
7 Conclusion
CrisiLangs extends CrisiText with human-translated warning messages in four additional languages. Fine-tuning improves automatic metrics across languages and configurations, while human preferences favor low temperatures and show no consistent benefit from guidelines.
7 Conclusion
- CrisiLangs contains 5,000 event-message pairs: 1,000 events from 411 scenarios, with warning messages in Italian, Bulgarian, German, Greek, and English.The four additional language versions are human-translated.Source: §7 · p. 9
- Fine-tuning improved ROUGE-L, BLEU, and BERTScore over the zero-shot baseline in every language and prompt configuration.The comparison covers Llama-3.1-8B-Instruct fine-tuned on five languages, with and without emergency guidelines.Source: §7 · p. 9
- Temperature 0.2 received the most human preferences in all five languages and both prompt configurations.Lower decoding temperatures also yielded higher automatic scores.Source: §7 · p. 9
- Human preferences did not show a reliable benefit from guidelines: four annotators preferred the plain configuration, while the Greek annotator preferred guidelines.For Bulgarian, automatic metrics were close between configurations even though no candidate was acceptable for 13 of 50 items.Source: §7 · p. 9
Limitations
The evaluated setup translates event inputs into the target language before generating warnings, and does not test direct English-input generation. The study also uses one multilingual model and a small expert-reviewed multilingual subset, with translation noise remaining on the input side.
Limitations
- The evaluated pipeline translates events into the target language before generating warnings; direct English-event input with target-language output is supported by the resource but unevaluated.Evaluating that single-pass setting would require another round of automatic and human evaluation.Source: Limitations · p. 9
- The study fine-tunes only one multilingual model, leaving language-specific model comparisons and English-only fine-tuning comparisons for future work.The authors also note that one native speaker per language handled translation and preference annotation, so inter-annotator agreement remains unmeasured.Source: Limitations · p. 9
- CrisiLangs has 1,000 expert-reviewed pairs per language versus more than 100,000 English pairs in CrisiText, and translation noise remains in event and guideline inputs.Only warning messages were human-translated; events and guidelines serve as inputs.Source: Limitations · p. 9
A Hyperparameters and Generation Settings
The appendix reports the adapter, supervised fine-tuning, prompt, and generation settings used for the experiments. The zero-shot baseline uses the same base model and decoding setup without the adapter.
A Hyperparameters and Generation Settings
- The fine-tuned Llama-3.1-8B-Instruct used a LoRA adapter with rank 16 and alpha 16, targeting attention and MLP projection layers.The adapter used dropout 0 and no bias adaptation; base weights were loaded in bfloat16 without quantization, then merged with the adapter before generation.Source: A Hyperparameters and Generation… · p. 11
- Supervised fine-tuning used up to 10 epochs, effective batch size 8, learning rate 3e−4, cosine scheduling, and early stopping on validation loss.Training used SFTTrainer, sequences capped at 1024 tokens, and the checkpoint with the lowest validation loss; the two configurations shared hyperparameters and 4,000 training examples.Source: A Hyperparameters and Generation… · p. 11
- The prompt configurations differed only in whether localized emergency-guideline blocks accompanied the instruction and event chain.Turns used the Llama-3.1 chat template with right padding.Source: A Hyperparameters and Generation… · p. 11
- Each test prompt was sampled once at five temperatures, with top-p 1 and up to 128 new tokens; the zero-shot baseline used the same model without LoRA and the same decoding setup.Each configuration produced 2,500 generations; runs used one NVIDIA A40 GPU.Source: A Hyperparameters and Generation… · p. 11
B Automatic Metrics
The appendix specifies the automatic metrics and their language-sensitive scoring choices. It also reports per-language BLEU components and generation lengths for the T=0.2 comparison.
B Automatic Metrics
- ROUGE-L was computed as message-level longest-common-subsequence F1, using a Unicode-aware tokenizer to retain Bulgarian and Greek characters.The default tokenizer would retain only about 2% of Bulgarian and 5% of Greek reference tokens, mostly digits and Latin-script names; scores were medians of 1,000 bootstrap resamples.Source: B Automatic Metrics · p. 11
- BLEU used Sacre-BLEU’s default corpus-level settings and 13a tokenizer, with scores divided by 100 to match the other metrics’ 0–1 scale.Reported scores were computed on CPU from released generation files.Source: B Automatic Metrics · p. 11
- BERTScore used xlm-roberta-large across all five languages without IDF weighting or baseline rescaling; its unrescaled scores should be compared only within their own columns.No rescaling baseline was available for Bulgarian and Greek; each score is the mean of per-item F1 scores.Source: B Automatic Metrics · p. 11
- In all five languages and both prompt configurations, fine-tuned generations were closer to reference length and had higher n-gram precision than the corresponding baseline.Table 6 reports per-language mean generation length and BLEU without the brevity penalty at T=0.2; comparisons are within language.Source: B Automatic Metrics · p. 11, Table 6 · p. 12
C Human Annotations
Crisis-management organizations supplied native-speaking annotators selected for professional experience, and the study used blinded human comparisons of model outputs across languages and configurations. Figure 4 reproduces the annotator instructions.
C Human Annotations
- Annotators were native speakers from crisis-management organizations, selected for professional experience and compensated fairly under applicable national laws.Source: C Human Annotations · p. 12
- For each language and model configuration, annotators blindly chose among outputs from temperatures 0.6, 0.4, and 0.2 on 50 test events.Candidates were labeled A/B/C; annotators could optionally leave a free-text comment.Source: C Human Annotations · p. 12
- Figure 4 reproduces the instructions for both translation and preference annotation.Translation instructions accompanied the English source; preference instructions accompanied a spreadsheet for each training configuration.Source: C Human Annotations · p. 12, Figure 4 · p. 12