Source-linked AI summary
ROBE: Reversed-Order-Biased-Experts for Extracting Extreme Long-tail Events from Historical Texts
Stella Verkijk, Piek Vossen
TL;DR
Extreme long-tail event extraction is difficult in niche, low-resource historical text, especially when event labels are scarce and unevenly distributed. The paper proposes ROBE, which prioritizes experts for underrepresented event groups and uses domain-specific synthetic data; ROBE improves precision, while ROBE+ improves both recall and precision over a general fine-tuned model.
Problem
The paper addresses event extraction from niche Early Modern Dutch VOC archives, where 70 labels range from 794 to 1 training examples and long-tail events are scarce.
Method
ROBE trains experts on frequency- or semantically grouped event classes, prioritizes experts for underrepresented labels, and supplements training with domain-specific synthetic data.
Results
ROBE improves precision over the all-class fine-tuned baseline, while ROBE+ improves both recall and precision; synthetic data improves precision but not recall.
Takeaways & Limitations
Combining prioritized classifiers can improve performance across classes and balance precision and recall in skewed, low-resource datasets.
Takeaways & Limitations
External validation assessed precision but not recall, and the paper calls for deeper synthetic-data and model-error analyses.
Abstract
from arXiv · showhide
This paper proposes methods to extract over 50 types of events from a Dutch historical corpus spanning the 17th and 18th centuries. The methods we propose aim to tackle the impossible: extracting the long-tail of the long-tail. Historic data from before the 19th century is in itself a niche domain not covered in the pre-training of Large Language Models, and we aim to extract events only very scarcely annotated in the training data available for this domain. We propose creating expert classifiers for subgroups of the events present in the training data. We make these groupings based on similar frequency in the training data or on semantic relatedness. Experts trained on underrepresented events are assigned higher priority when predicting to avoid being dominated by frequency biases. We refer to this new way of combining classifiers, specifically tailored to protect the long-tail, as ROBE: Reversed-Order-Biased-Experts. We also propose a controlled method to create domain-specific synthetic data. Our two implementations of ROBE outperform a simple fine-tuned encoder model with a .10 increase in recall and a .16 increase in precision respectively. The best model achieves a .10 increase in f1 for a group of long-tail classes in our niche data set.
1. Introduction
The paper targets extreme long-tail event extraction in skewed, niche VOC archives by combining prioritized expert classifiers and domain-specific synthetic data. ROBE-based approaches improve precision, while ROBE+ improves both recall and precision over a general fine-tuned model.
- The study extracts events from Early Modern Dutch VOC archives with a highly skewed training distribution.The corpus is niche and historical, making its event information especially long-tail.
- The task uses multiclass token classification with 70 event labels ranging from frequent to extremely scarce classes.Expert models are trained on groups formed by similar label frequencies or semantic relatedness.
- ROBE assigns higher prediction priority to experts trained on underrepresented labels, overriding predictions from experts representing more frequent labels.This reverses the usual frequency bias in classifier combination.
- Previous encoders failed to capture relevant VOC-archive semantics even after fine-tuning, motivating use of GloBERTise-based implementations.GloBERTise was pretrained solely on VOC archives.
- ROBE improves precision over the all-class baseline, while ROBE+ improves both recall and precision.Synthetic data improves precision but not recall.
- The approach offers a low-resource solution that can mitigate underperformance on long-tail knowledge without additional domain-specific resources or large generative models.The paper also reports that splitting labels into groups improves performance more than synthetic data alone.
2. Related Work
Prior work covers historical event-related extraction, classifier combination, and synthetic data, but ROBE combines these strands through frequency- and semantics-based experts with fixed priority ordering. Its synthetic-data method injects domain-specific examples into prompts without fine-tuning the generative model.
- Historical event extraction: Historical extraction studies address event attributes or smaller event-type inventories, whereas this work classifies events in a broader historical setting.BRAD, for example, contains 12 event types and nineteenth-century African American periodicals.
- Historical event extraction: Generative language models have underperformed task-specific systems in long-tail event extraction and shown low precision when extracting entities from historical reports.These findings motivate specialized approaches for niche historical extraction.
- Combining classifiers: ROBE differs from standard ensembling by overriding predictions from classifiers that encode stronger frequency biases rather than weighting all classifiers equally.It resembles a mixture of experts but uses a fixed rule-based classifier order instead of an extra selection algorithm.
- Synthetic data generation: Synthetic data generation can reduce laborious annotation and curation, and generative language models have expanded its use in low-resource NLP.Research also evaluates generation quality, generated data, and prompting methods.
- Synthetic data generation: The paper creates domain-specific synthetic data by adding data points and annotation examples to prompt templates without fine-tuning or instructing the generative model.This steers training toward long-tail phenomena.
3. Data
The dataset consists of GLOBALISE VOC archival scans and uses an event-classification task in which any part of speech may refer to an event. Its labels are extremely imbalanced, with violence-related events concentrated in the long tail.
- The training data contains 280 scans from 26 documents, while the test data contains 18 scans from 3 documents.The data was created and released by the GLOBALISE project.
- Unlike predicate-centered event extraction, the task allows any part of speech to refer to an event.Long sentences with unclear boundaries and implicit event information motivate this design.
- The label distribution is Zipfian: Communication occurs 794 times, while ScalarChange occurs once.Communication represents 16% of event-labeled tokens, ScalarChange 0.02%, and 92% of tokens have no event label.
- VOC archives emphasize trade and politics, while violence-related events such as Mutiny, Besieging, Invasion, and Enslaving are much less frequent.This distribution reflects a cultural bias in the data and risks underrepresenting VOC violence.
4. Method
ROBE trains expert classifiers on grouped event labels to reduce domination by frequent classes, and supplements this with controlled, domain-specific synthetic data. The evaluation compares these approaches with fine-tuned and baseline systems on adjudicated and non-adjudicated test sets.
- 4.1. The ROBE approach: ROBE groups event classes by similar training frequency or semantic relatedness, training separate experts to reduce frequent classes overshadowing scarce ones.Grouping is intended to help models converge for classes with similar representation; excluded labels are relabeled as O during group training.
- 4.1. The ROBE approach: Experts are prioritized linearly, with experts trained on the least represented labels receiving highest priority when predictions disagree.The highest-priority expert determines the selected label in a disagreement, while the most represented group receives lowest priority.
- 4.1. The ROBE approach: ROBE expert groups may merge related classes, including dynamic–static pairs, shared taxonomy subclasses, or events one taxonomy step apart.Examples include Dying with BeingDead, Destroying with BeingDestroyed, and Besieging with Invasion.
- 4.2. Synthetic data creation: Synthetic data prompts combine domain-specific entities and definitions with annotated examples to generate Early Modern Dutch event sentences.The prompts target archival style and mark the event-referring word in square brackets; 375 annotated sentences were generated for Translocation and Transportation.
- 4.3. Overview of evaluated methods: The evaluation compares EtE, synthetic-data-augmented fine-tuning, ROBE, ROBE+, a lexicon, and GenLLM against the test set.ROBE+ includes EtE as the lowest-priority expert; the test data contains three documents, with annotations adjudicated through majority voting, guidelines, or historian consultation.
- 4.4. Evaluation: Because annotations are partially subjective, performance is evaluated on both adjudicated and non-adjudicated test sets.For the non-adjudicated set, a prediction counts as correct when it matches any one of the gold labels.
5. Results
The evaluation compares ROBE variants, fine-tuned baselines, and a lexical baseline under strict overall scoring and grouped translocation or violence scoring. ROBE improves precision, ROBE+ generally leads recall and F1, and external validation provides additional precision-focused evidence.
- 5.1. Evaluation on test set: ROBE scores up to .16 higher in precision than EtE, while recall increases slightly for challenging violence, repression, and conflict events.Overall recall decreases, but the long-tail event subset shows a slight recall increase.
- 5.1. Evaluation on test set: ROBE+ scores highest in recall and f1 in most evaluation settings by combining expert classifiers with a general fine-tuned model.The combined model balances the experts’ precision advantage with the general model’s broader training coverage.
- 5.1. Evaluation on test set: EtE+synth improves precision over EtE but declines in recall, while substituting a synthetic translocation expert does not increase ROBE or ROBE+ scores.The authors attribute the recall decline to unlabelled mentions of other events acting as negative examples.
- 5.1. Evaluation on test set: The lexical baseline scores highest in precision across the board, whereas GPT5.1 performs poorly overall and unreliably on long-tail classes.These comparisons are reported alongside the ROBE variants in the test-set evaluation.
- 5.2. External Validation: In workshop validation, ROBE predictions were judged useful 75% of the time in the first iteration and 79% in the second.The first iteration covered 329 predictions with six validators; the second covered 578 predictions with nine participants.
- 5.2. External Validation: External validation measures precision only because validators judged predicted events but could not assess events ROBE missed.The authors report that this validation produced higher precision than the classical evaluation and contextualized the Table 3 scores.
6. Discussion
The discussion presents ROBE as an accessible approach for skewed, low-resource, niche-domain event classification. It also identifies unresolved questions about synthetic data and model-specific errors.
- 6. Discussion: ROBE combines classifiers with a fixed priority order that favors experts trained on underrepresented labels to improve performance across classes.The approach is intended to balance precision and recall in a skewed dataset.
- 6. Discussion: The approach is presented as accessible for long-tail event classification in a low-resource language and niche domain.The authors state that it requires little to no extra resources or complicated engineering.
- 6. Discussion: Analyzing the synthetic data could clarify why it improves precision but not recall.This is identified as one limitation of the current study.
- 6. Discussion: A finer-grained error analysis comparing model predictions on tokens referring to specific long-tail events would clarify differences between configurations.The authors propose this analysis as a further way to understand model behavior.