PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement

Rafael Macalaba, Aivin V. Solatorio, Patrick Michael Brock, Olivier Dupriez

arXiv:2609.12107v1cs.CLcs.AIcs.IRecon.EM

TL;DR

Dataset references in FCV and forced-displacement documents are difficult to track because they are dispersed and inconsistently cited. The paper adapts a lightweight extractor using LLM-refined weak labels and targeted synthetic examples; on an independent benchmark, it reports 89.5% precision on positive passages and 88.16% accuracy overall. The study evaluates extraction, not the full displacement data landscape.

  • Problem

    Dataset references are dispersed across unstructured documents and inconsistently cited, making systematic tracking difficult.

    Source: §1 · p. 2

  • Method

    A lightweight model generates candidate references that an LLM reviews and corrects, with targeted synthetic examples supplementing supervision for fine-tuning.

    Source: §3.1 · p. 5, §3.2 · p. 5

  • Results

    89.5% precision on positive passages and 88.16% accuracy across the complete holdout benchmark; specificity is 88.6%.

    Source: §5.4 · p. 12, §6 · p. 12

  • Takeaways & Limitations

    The framework provides a technical component for extracting references across heterogeneous research, humanitarian, and operational documents, while difficult negative cases remain a challenge.

    Source: §7 · p. 14

    Dataset mentions alone cannot establish whether data were unavailable, unused, or insufficient; broader landscape analysis requires representative corpus construction and additional classification and linkage steps.

    Source: §6.3 · p. 13

Abstract

from arXiv · show

Development and humanitarian organizations produce and support surveys, administrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced remains difficult. Such references are dispersed across research papers, project documents, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemination. We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction. We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1\% precision and 70.5\% recall at the mention level; among passages containing dataset references, precision reaches 89.5\%. At the passage level, the model achieves 88.2\% accuracy and 88.6\% specificity in distinguishing passages with dataset references from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape.

1 INTRODUCTION

The introduction motivates scalable dataset-reference tracking in FCV and forced-displacement documents, where references are inconsistently cited and domain-specific labels are scarce. It presents weakly supervised adaptation and an independent heterogeneous benchmark, while distinguishing extraction evaluation from full data-landscape analysis.

1 INTRODUCTION

  • In FCV and forced-displacement settings, inconsistent dataset citations make it difficult to track references dispersed across research, policy, project, and humanitarian documents.Organizations may use these patterns to assess data reach and, alongside information about available data and coverage, identify potential production, accessibility, or dissemination gaps.Source: §1 · p. 2
  • Domain-specific labeled data are scarce, and models trained on research literature may not transfer directly to specialized FCV and forced-displacement documents.Constructing a large training corpus manually would require substantial expert annotation.Source: §1 · p. 2
  • The framework uses a research-literature extractor to generate candidates, an LLM to review them in context and correct boundaries, and the resulting annotations to fine-tune a lightweight domain model.Targeted synthetic and contrastive examples supplement the annotations.Source: §1 · p. 2
  • The resulting model is evaluated on an independent gold-standard benchmark of 1,706 passages spanning research, humanitarian, and operational documents.The paper presents the framework, its transfer to a lightweight local model, and evaluation on a heterogeneous benchmark as its contributions.Source: §1 · p. 2
  • The study evaluates extraction capability rather than the full displacement data landscape, and its benchmark tests transfer across document types rather than representing all relevant materials.Broader analysis would additionally require systematic corpus construction, displacement-relevance filtering, dataset linkage, and classification of how identified data are used.Source: §1 · pp. 2–3

2 RELATED WORK

Related work situates the paper at the intersection of inconsistent dataset citation, automated mention extraction, weak supervision, and data-use priorities in forced-displacement and FCV settings.

2.1 DATASET CITATION AND DISCOVERABILITY

  • Inconsistent dataset citation practices impede discoverability; this paper extracts references as they appear in unstructured text rather than proposing a citation standard.Prior work discusses citation conventions, citation graphs, and persistent identifiers (Silvello, 2018).Source: §2.1 · p. 3

2.2 AUTOMATED DATASET MENTION EXTRACTION

  • Prior dataset-mention systems use NER and question-answering, while this paper adapts a research-literature extractor to humanitarian, operational, and FCV documents where references are sparser and less standardized.Younes & Scherp (2023) found question-answering formulations generalized better to unseen dataset names; the paper uses an LLM for candidate adjudication rather than a fixed tagger.Source: §2.2 · p. 3

2.3 WEAK SUPERVISION AND LLM-ASSISTED ANNOTATION

  • The pipeline follows weak supervision by generating candidates with one upstream model and having a frontier LLM adjudicate them individually in context, while retaining human-constructed gold data for final evaluation.The authors relate this design to Ratner et al. (2017), LLM-based data programming, self-training, and active learning (Wang et al., 2022).Source: §2.3 · pp. 3–4

2.4 DATA USE IN FORCED DISPLACEMENT AND FCV SETTINGS

  • Strategic priorities emphasize producing, accessing, and using data in displacement and FCV settings, while systematic visibility into references may help characterize data reach and potential gaps.Such gap assessment requires combining observed use and non-use with information about data availability, populations, and topics covered.Source: §2.4 · p. 4

3 METHODOLOGY

The methodology transfers supervision from unlabeled FCV and forced-displacement documents into a lightweight extractor through candidate generation, LLM refinement, and targeted synthetic examples. The fine-tuned local model can then extract at scale without LLM verification during inference.

3 METHODOLOGY

  • The four-stage framework generates candidate mentions with a lightweight model, refines them with an LLM, expands training examples synthetically, and fine-tunes the lightweight model for extraction.This separates training-data construction from inference: the LLM improves annotations selectively, while the local model can later process collections independently.Source: §3 · pp. 4–5

3.1 WEAKLY SUPERVISED CANDIDATE GENERATION

  • The pretrained extractor locates possible references in unlabeled documents as weak annotations that narrow the review search space but are not assumed correct.Candidate coverage is prioritized with a permissive threshold of τ = 0.15, and the initial model acts as a screening mechanism rather than a final annotator.Source: §3.1 · p. 5

3.2 LLM-ASSISTED ANNOTATION REFINEMENT

  • The context-aware LLM validates whether candidates are structured data resources and corrects span boundaries, excluding items such as projects or narrative publications.Because refinement only considers surfaced candidates, pipeline recall depends on the initial candidate generator's coverage.Source: §3.2 · pp. 5–6

3.3 ANNOTATION SCHEMA

  • The annotation schema distinguishes named, descriptive, and vague dataset references, and supports metadata such as acronym, publisher, geography, population, and year when available.These metadata fields are not part of the extraction evaluation reported in the paper.Source: §3.3 · p. 6

3.4 TARGETED SYNTHETIC DATA GENERATION

  • Synthetic examples broaden coverage of dataset types, linguistic forms, and underrepresented cases rather than reproducing the source corpus distribution.Few-shot prompts use selected dataset names and contexts, with more than 100 templates spanning six subject domains; generated records undergo schema and automated validation.Source: §3.4 · pp. 6–7

3.4.1 CONTRASTIVE EXAMPLES

  • Contrastive examples teach context-sensitive labels for similar expressions, such as extracting LSMS as a survey but not as an LSMS fieldwork team.The examples distinguish data resources from data-collection activities; Table 1 illustrates contrastive synthetic examples.Source: §3.4.1 · p. 7, Table 1 · p. 7

3.5 FINE-TUNING AND FINAL EXTRACTION

  • Real-document annotations and targeted synthetic examples are combined to fine-tune GLiNER2, which is then applied at τ = 0.40 without LLM verification at inference.Performance is evaluated on a separate gold-standard holdout of 1,706 text chunks not used to construct annotations, generate examples, or fine-tune the model.Source: §3.5 · p. 7

4 DATA AND EVALUATION DESIGN

The evaluation corpus is designed to test transfer across heterogeneous document types, not to represent displacement data use. It assesses both dataset-mention extraction and passage-level identification of references.

4 DATA AND EVALUATION DESIGN

  • The corpus is a technical validation set, not a representative sample, and does not estimate data-use prevalence or distribution across research, policy, and operations.A broader landscape map would require a separately designed corpus covering the relevant evidence ecosystem and identifying displacement-relevant passages in broader documents.Source: §4 · p. 8

4.1 DOCUMENT COLLECTIONS

  • The 93-document corpus spans UNHCR/ReliefWeb reports (43), SEIS and briefs (25), policy research papers (14), and project appraisal documents (11).Candidates were LLM-judged, human-adjudicated, and filtered by agreement, with document-disjoint splits.Source: §4.1 · pp. 8–9

4.2 GOLD-STANDARD EVALUATION SET

  • The holdout contains 1,706 chunks: 545 with dataset references and 1,161 without; prevalence ranges from about 39% in SEIS/baseline passages to about 5% in PAD passages.The benchmark includes both data-rich and data-sparse conditions.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 EVALUATION METRICS

  • Evaluation at the mention and passage levels tests whether the model locates dataset references and distinguishes passages with references from those without.The final model is evaluated at prediction confidence threshold τ = 0.40.Source: §4.3.1 · p. 9, §4.3.3 · p. 10

4.3.1 DATASET-MENTION EXTRACTION

  • A mention counts as correct when its token-span Jaccard overlap with the annotation is at least 0.5; precision, recall, and F0.5 are reported.F0.5 gives greater weight to precision, which matters because false positives could distort estimates of observed use and apparent gaps.Source: §4.3.1 · p. 9

4.3.2 PERFORMANCE ON PASSAGES WITH AND WITHOUT DATASET MENTIONS

  • Mention extraction is reported across the full holdout and separately on passages containing at least one annotated dataset reference, separating false predictions from missed references.The first setting reflects processing complete document collections; the second isolates identification when datasets are present.Source: §4.3.2 · p. 9

4.3.3 PASSAGE-LEVEL CLASSIFICATION

  • Passage-level classification predicts positive when at least one dataset mention is extracted; accuracy, recall, F1, and specificity are reported.Specificity measures correct negative predictions and is particularly important where dataset mentions are sparse amid operational content.Source: §4.3.3 · p. 10

5 RESULTS

On the 1,706-passage benchmark, the model extracts dataset mentions with 74.1% precision and 70.5% recall across all passages, and detects dataset-containing passages with 88.16% accuracy. Results are consistent across several document collections, while policy-corpus transfer remains untested.

5.1 DATASET MENTION EXTRACTION

  • 74.1% precision and 70.5% recall across the full benchmark, with 89.5% precision on passages containing dataset references.Table 3 reports results at τ = 0.40.Source: Table 3 · p. 10, §5.1 · p. 10

5.3 PERFORMANCE ACROSS DOCUMENT TYPES

  • Positive-passage precision stays near 90% for PRWPs, UNHCR/ReliefWeb documents, and the SEIS/baseline collection; specificity exceeds 86% in every collection.PAD precision is 100% on positive passages, but that subset contains few annotated mentions.Source: §5.1 · p. 10, §5.3 · p. 11

5.2 DATASET DETECTION

  • 88.16% accuracy, 87.16% recall, and 88.63% specificity for passage-level dataset detection across the full benchmark.The benchmark includes 1,161 passages without dataset references.Source: Table 4 · p. 11, §5.2 · p. 11

5.3 PERFORMANCE ACROSS DOCUMENT TYPES

  • The benchmark does not establish transfer to government strategies, national policy documents, or other policy-oriented corpora; direct evaluation there remains necessary.Its document diversity offers some evidence that extraction is not tied to one genre, but only within the represented document types.Source: §5.3 · p. 11
  • At τ = 0.15, LLM refinement rejected 145 of 203 PAD candidates, most often project or program titles (50.3%) and administrative software or monitoring systems (17.2%).These dataset-adjacent entities are recurring sources of ambiguity in operational documents, motivating contrastive negative examples and further training-data expansion.Source: §5.3 · p. 12

5.4 SUMMARY OF EVALUATION RESULTS

  • The evaluation supports adapting a general extractor without a large manually labeled corpus, with 89.5% positive-passage precision and 88.16% passage-level accuracy.Mention-level recall of approximately 70% leaves room to expand weak-supervision candidate coverage; operational documents may benefit from additional contrastive examples.Source: §5.4 · p. 12

6 DISCUSSION

The discussion presents the framework as a way to adapt dataset extraction with limited labeled data, while emphasizing that distinguishing datasets from similar entities is central to the task. It frames extraction as one component of a broader data-landscape analysis, which requires additional evidence and validation.

6 DISCUSSION

  • 89.5% precision on positive passages and 88.6% specificity show strong results on dataset-bearing passages and the ability to distinguish them from other content.The discussion notes that most benchmark passages contain no dataset reference, making correct non-extraction part of the task.Source: §6 · p. 12

6.1 ADAPTING EXTRACTION MODELS WITH LIMITED LABELED DATA

  • Weak candidate labels are refined by an LLM and supplemented with synthetic examples before fine-tuning a lightweight model for scalable extraction.The workflow shifts expert effort from exhaustive annotation toward annotation rules, difficult error cases, and a rigorous gold-standard evaluation set.Source: §6.1 · pp. 12–13

6.2 LEARNING WHAT NOT TO EXTRACT

  • Because operational documents contain dataset-like project titles and systems, continued gains may depend on adding negative and contrastive examples as well as positive mentions.The discussion identifies such near-miss entities as a substantial share of filtered candidates during refinement.Source: §6.2 · p. 13

6.3 FROM DATASET DETECTION TO MAPPING THE DISPLACEMENT DATA LANDSCAPE

  • Mention patterns may help trace data use or identify potential production gaps, but mentions alone cannot establish dataset availability, use, or sufficiency.Broader interpretation requires representative corpora, displacement-relevance and use-context classification, dataset linkage, and comparison with data inventories and information needs.Source: §6.3 · p. 13
  • Dataset relevance must be judged from the context of a reference, since a dataset’s coverage does not establish that it was used for displacement-related analysis.A national survey might include displaced populations yet be cited for unrelated national findings, or be used specifically to analyze displaced populations.Source: §6.3 · p. 13

6.4 FUTURE DIRECTIONS

  • Future work includes targeted ablations, broader candidate generation, expert-agreement checks, expanded operational evaluation, and testing transfer to policy-oriented corpora.The discussion also proposes integrating corpus construction, dataset resolution, use-context classification, and available-data inventories into a broader pipeline.Source: §6.4 · p. 14

7 CONCLUSION

The conclusion presents a weakly supervised approach for extracting dataset references from FCV and forced-displacement documents without a large manually labeled corpus. Its benchmark evaluation supports use across heterogeneous document types, while broader landscape analysis would require additional components.

7 CONCLUSION

  • The framework combines candidate generation, LLM-assisted annotation refinement, synthetic and contrastive examples, and fine-tuning of a lightweight local model.It is designed to adapt dataset extraction without requiring a large manually annotated training corpus.Source: §7 · p. 14
  • The model identifies references across research, humanitarian, and operational documents, while distinguishing dataset mentions from similar entities such as project titles and administrative systems.The candidate-refinement analysis suggests difficult negative cases are important and motivates targeted ablation studies.Source: §7 · p. 14
  • Scalable extraction is one component of a broader analysis that could examine data reach, barriers to uptake, and potential production gaps when combined with corpus, linkage, context, and inventory information.The paper presents this as a possible extension beyond monitoring the downstream reach of existing data investments.Source: §7 · pp. 14–15

OPEN SOURCE, ETHICS, DISCLOSURE, AND DISCLAIMER

The paper reports public availability of its fine-tuned model and supporting materials, describes its document-level data handling, discloses AI use, and clarifies that its findings are the authors’ own.

OPEN SOURCE, ETHICS, DISCLOSURE, AND DISCLAIMER

  • The fine-tuned extraction model, model card, inference examples, and supporting documentation are publicly available.The paper provides a Hugging Face collection link for these materials.Source: OPEN SOURCE, ETHICS, DISCLOSURE, AND… · p. 15
  • The study analyzes publicly available document text, not individual-level records or personally identifiable information.The paper says human review remains appropriate when automated outputs inform consequential monitoring, reporting, or evaluation.Source: OPEN SOURCE, ETHICS, DISCLOSURE, AND… · p. 15
  • AI tools were used for candidate verification, synthetic-data generation, model development, and manuscript editing; methodological decisions and conclusions remain the authors’ responsibility.Source: OPEN SOURCE, ETHICS, DISCLOSURE, AND… · p. 15
  • The findings and conclusions are the authors’ and do not necessarily represent the views of the World Bank, its affiliated organizations, Executive Directors, or represented governments.Source: OPEN SOURCE, ETHICS, DISCLOSURE, AND… · p. 15

A.1 A. LLM-ASSISTED ANNOTATION REFINEMENT

Candidate spans are reviewed by GPT-4o with surrounding context, and returned annotations are constrained by a response schema and rules in a system prompt.

A.1 A. LLM-ASSISTED ANNOTATION REFINEMENT

  • The weakly supervised candidate generator’s dataset spans are sent to GPT-4o with surrounding context for validity decisions.The model returns its decision through OpenAI’s structured outputs API.Source: §A.1 · p. 17
  • A Pydantic response schema constrains the structure of the returned annotation.Source: §A.1 · p. 17
  • A system prompt defines valid dataset references and rules for distinguishing datasets from other entities.Source: §A.1 · p. 17

A.1.1 PYDANTIC RESPONSE FORMAT

The Pydantic response schema structures the LLM’s annotation-refinement output so its decisions and annotations can be parsed and used in training-data construction.

A.1.1 PYDANTIC RESPONSE FORMAT

  • The Pydantic schema structures the LLM’s validation decisions and annotations for consistent parsing and incorporation into training-data construction.Figure 1 presents the response schema used for LLM-assisted annotation refinement.Source: §A.1.1 · p. 17, Figure 1 · p. 17

A.1.2 LLM JUDGE SYSTEM PROMPT

The appendix specifies how the LLM judge identifies valid data mentions, how synthetic examples are generated and checked, and how extraction is evaluated. It also illustrates why broad candidate generation requires context-aware refinement.

A.1.2 LLM JUDGE SYSTEM PROMPT

  • A valid data mention refers directly to a structured data resource, including vague generic references, while specified author attributions, entities, methods, publications, and layout artifacts are excluded.The judge also assigns named, descriptive, or vague specificity according to whether the resource has a proper name or identifying details.Source: §A.1.2 · p. 18

A.2 B. SYNTHETIC DATA GENERATION; A.2.1 DOMAIN VARIABLE BANKS

  • Six domain-specific banks steer synthetic examples toward dataset names and related resources relevant to development, humanitarian, and FCV applications.Synthetic examples supplement annotations refined from real documents.Source: §A.2.1 · p. 18, §A.2 · p. 18, Table 5 · p. 19

A.2.2 VALIDATION OF SYNTHETIC EXAMPLES

  • Synthetic examples are retained only when requested dataset and metadata attributes appear verbatim, spans exclude standalone years, and spans do not end in incomplete fragments.Examples that fail any check are excluded from the training corpus.Source: §A.2.2 · p. 19

A.2.3 CONTRASTIVE EXAMPLES

  • Contrastive examples are designed to reduce reliance on capitalization, acronyms, or familiar terms by requiring context to indicate that an expression refers to a data resource.Table 6 provides illustrative contrastive examples.Source: §A.2.3 · p. 19, Table 6 · p. 19

A.3 C. ADDITIONAL EVALUATION DETAILS; A.3.1 MENTION-LEVEL SPAN MATCHING

  • Mention-level evaluation compares predicted and gold spans using token-level Jaccard similarity, allowing minor boundary differences while requiring substantial overlap.The appendix defines the predicted and gold token sets used for the comparison.Source: §A.3.1 · p. 20

A.3.2 PASSAGE-LEVEL DETECTION

  • Passage-level evaluation labels a passage positive when at least one dataset mention is extracted and negative when none is returned.This tests discrimination between dataset-bearing and non-dataset-bearing passages.Source: §A.3.2 · p. 20

A.4 D. ANALYSIS OF FILTERED CANDIDATE MENTIONS

  • Filtered candidates in FCV operational documents are often meaningful project titles, administrative systems, or monitoring tools that resemble datasets.The framework uses broad candidate generation followed by refinement to determine whether expressions refer to structured data resources.Source: §A.4 · pp. 20–21, Table 7 · p. 21

A.4.1 ILLUSTRATIVE CONTEXT EXAMPLES

  • Candidate strings alone can be ambiguous: terms such as “Survey” or “Assessment” signal possibilities, but refinement evaluates each candidate’s role in surrounding text.Table 8 illustrates contexts for valid references and filtered candidates.Source: §A.4.1 · pp. 21–22, Table 8 · p. 22
Loading 2609.12107v1…