PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLM-assisted regularization increase forecast accuracy for migration flows in low data regimes?

Nathaniel T. Hindman, Fabricio Murai

arXiv:2610.07208v1cs.LGcs.CY

TL;DR

Structured migration indicators can be delayed or unavailable even as migration conditions change rapidly. This paper uses hierarchical LLM pipelines to extract news-derived signals for gravity models and weighted Lasso; forecasting results vary across corridors and strategies, with no single approach consistently best.

  • Problem

    Structured socio-economic indicators can be infrequent or delayed, and conflict can constrain data availability even as migration dynamics change rapidly.

    Source: §I · p. 1

  • Method

    Hierarchical LLM pipelines classify news articles for migration signals, which inform push–pull features and feature-specific penalties in weighted Lasso forecasting models.

    Source: §I · p. 2, A. Proposed features: push-pull count… · p. 4, D. Extracting Push-Pull Factors from… · p. 5

  • Results

    No regularization strategy consistently outperformed the others; the Mexico push–pull-ratio gravity configuration achieved a MAPE of 17.15%.

    Source: Impact of number of articles · p. 9, §VI · p. 10

  • Takeaways & Limitations

    News-derived context can improve migration forecasting in some settings when incorporated through contextual signal extraction and structured feature weighting.

    Source: §I · p. 2, §VI · p. 10

    Increasing article volume alone did not consistently improve accuracy and often introduced variance and instability.

    Source: §VI · p. 10

Abstract

from arXiv · show

Predicting migration flows remains a significant challenge for traditional gravity-based forecasting models, which primarily rely on structured socio-economic indicators such as economic disparity, political stability, and geographic distance. This work investigates whether Large Language Models (LLMs) can improve migration forecasting by extracting contextual migration-related signals from news articles and incorporating them into a weighted Lasso forecasting framework through feature-specific regularization penalties. The proposed framework uses hierarchical LLM inference pipelines to classify migration-related push--pull signals from news data and evaluates the resulting forecasting performance across multiple migration corridors between November 2021 and November 2022, including Mexico--United States, Ukraine--Poland, and Syria--Turkey. Experimental results showed mixed performance across migration corridors and modeling strategies, and no single regularization approach consistently outperformed the others across all experiments. The best-performing Mexico configuration, which consisted of a gravity-based model augmented with the proposed push--pull ratios, achieved a Mean Absolute Percentage Error (MAPE) of 17.15%, while the strongest Syria configuration achieved a MAPE of 29.29% using Direct LLM-Lasso. For Ukraine, the best-performing configuration used LLM-Assisted Regularization (AR) and achieved a MAPE of 41.05%. Overall, the results suggest that contextual article-derived features and LLM-guided regularization can improve migration forecasting under certain conditions, although migration corridor characteristics, article volume, and hyperparameter configuration strongly influenced performance.

I. INTRODUCTION

Traditional migration forecasting relies on structured indicators that can lag rapidly changing conditions or be unavailable in conflict settings. This paper combines gravity-model features with news-derived signals and LLM-assisted feature weighting.

I. INTRODUCTION

  • Structured indicators are often infrequent or delayed, while conflict can disrupt data collection and directly observed migration-flow data is sparse.These constraints are especially consequential when migration conditions shift rapidly.Source: §I · p. 1
  • The framework combines gravity-model features with news-derived contextual signals to adjust feature importance through LLM-assisted classification and regularization.LLMs extract signals from articles rather than directly predicting migration flows.Source: §I · p. 2
  • The study evaluates hierarchical LLM pipelines for push–pull signal extraction and compares direct penalties with count-based feature weighting across three migration corridors.The corridors are Mexico–United States, Ukraine–Poland, and Syria–Turkey; experiments use monthly forecasting and MAPE.Source: §I · p. 2

A. Gravity Models of Migration

  • Gravity models relate migration flows to origin and destination populations and additional structural variables, with formulations tailored to particular country pairs.The related-work review also covers LLM use for L1 regularization and cross-modality forecasting with news text.Source: A. Gravity Models of Migration · pp. 2–3, §II · p. 2

B. L1 Regularization, LLM-Lasso and Feature Weighting

The paper uses news text as contextual evidence for migration forecasting, rather than as a direct flow predictor. It combines article-derived push–pull signals and semantic feature-category evidence with gravity models and weighted Lasso.

B. L1 Regularization, LLM-Lasso and Feature Weighting

  • Feature-specific Lasso penalties give different weights to coefficients, while directly generated LLM weights can be unstable.L1 regularization encourages sparse models by reducing less informative coefficients toward zero.Source: B. L1 Regularization, LLM-Lasso and… · p. 3

C. Incorporating News Data into Migration Forecasting

  • News articles can contain contextual migration signals, and the framework classifies them as push, pull, mixed, or neutral-like categories for forecasting use.The paper positions this approach alongside prior work using textual signals in migration forecasting.Source: C. Incorporating News Data into… · p. 3

III. PUSH-PULL FEATURES FOR GRAVITATION MODELS AND LLM-ASSISTED REGULARIZATION

  • The hybrid framework combines hierarchical LLM inference with weighted Lasso, using aggregated semantic evidence to inform feature-specific penalties.The resulting weights dynamically adjust predictor importance while retaining the interpretability of gravity models.Source: §III · pp. 3–4

A. Proposed features: push-pull count ratios

The framework combines structured socio-economic and geographic features with monthly news-derived push–pull ratios and uses article evidence to adjust feature-specific Lasso penalties. A hierarchical LLM pipeline classifies articles, escalating to larger models or full text when URL-level classification is uncertain.

A. Proposed features: push-pull count ratios

  • Monthly push, pull, mixed, and unclear ratios encode each category’s share of migration-related articles alongside structured country-level indicators.The model includes origin and destination indicators such as GDP, GNI, HDI, and GPI, plus geographic and corridor-specific factors.Source: A. Proposed features: push-pull count… · p. 4

B. The feature-augmented gravitational model

  • The feature-augmented gravity model combines structured indicators and news-derived ratios to model migration flow, with corridor-specific effects and residual error.Its listed predictors include origin and destination measures and the monthly push, pull, mixed, and unclear ratios.Source: B. The feature-augmented gravitational… · p. 4

C. Proposed method: LLM-Assisted Regularization

  • LLM-Assisted Regularization converts aggregated semantic evidence counts into inverse feature-specific penalties, reducing regularization pressure on frequently referenced features.The LLM identifies contextual evidence rather than directly generating numerical feature-importance scores.Source: C. Proposed method: LLM-Assisted… · pp. 4–5

D. Extracting Push-Pull Factors from News Media with LLMs

  • The hierarchical article-classification pipeline starts with lightweight URL-level inference and escalates uncertain cases to a larger model or full-article analysis.The configurations vary whether escalation adds model capacity, article context, or both.Source: D. Extracting Push-Pull Factors from… · p. 5

E. Forecasting Configurations

The study compares gravity forecasting with news-augmented features and LLM-derived feature penalties. It evaluates monthly corridor data and articles using rolling nine-month training windows.

E. Forecasting Configurations

  • Four configurations compare a gravity baseline with push–pull ratio augmentation, count-based LLM-Assisted Regularization, and Direct LLM-Lasso penalties.Gravity-PP uses normalized article ratios, while LLM-AR derives feature-specific penalties from aggregated semantic evidence.Source: E. Forecasting Configurations · pp. 5–6

F. Migration Flow Datasets

  • Monthly migration-flow observations cover the Mexico–United States, Ukraine–Poland, and Syria–Turkey corridors from November 2021 to November 2022.The observations come from the Meta International Migration Flows dataset.Source: F. Migration Flow Datasets · p. 6

G. News Datasets

  • The final news corpus contains migration-related articles grouped by month and corridor; earlier GDELT coverage was limited, prompting collection from mainstream news sources.The final dataset was assembled for Mexico–United States, Ukraine–Poland, and Syria–Turkey.Source: G. News Datasets · p. 6

H. Model Training

  • Models use sliding nine-month training windows to predict the next holdout month, incorporating article-derived features and feature-specific penalties once article data are available.Before article data became available, the framework used a gravity-based Lasso model with uniform regularization.Source: H. Model Training · p. 6

IV. EXPERIMENTS

The experiments assess hierarchical LLM pipelines for classifying migration-related articles and test article-derived features and regularization in migration forecasting. SuLuSa performed best overall in classification, with full-article context helping resolve uncertain cases.

A. Part I: Extracting Push–Pull Factors

B. Part II: LLM-based Regularization Approaches

  • Forecasting experiments used rolling monthly holdouts across three corridors to compare article-informed weighting and push–pull ratios, including different corpus sizes and Lasso strengths.The evaluation primarily used MAPE; larger corpora were tested, but intermediate corpus sizes were not extensively evaluated due to time and computational constraints.Source: Forecasting Experiments · p. 7

A. Part I results

  • SuLuSa achieved the lowest proportion of “unclear” classifications across most temperature settings and was selected for subsequent forecasting experiments.The authors attribute its advantage over URL-only SuLu to selectively adding model capacity and deeper article context.Source: A. Part I results · pp. 7–8
  • Temperature had minimal impact on classification behavior: Qwen SuLuSa classified 163 articles as “unclear” at τ = 0.3 and 159 at τ = 1.5.The constrained prompt and fixed classification schema are cited as likely contributors to consistency across temperatures.Source: Impact of Temperature and LLM Choice · p. 8

B. Part II Results

Forecasting results varied by corridor, with no regularization strategy consistently best; the strongest configurations differed across Mexico, Syria, and Ukraine. The conclusion also reports that hierarchical inference extracted push–pull signals, but article volume alone did not reliably improve forecasts.

B. Part II Results

  • No single regularization strategy consistently outperformed competing methods across migration corridors; the strongest approach depended on the migration setting and forecasting conditions.Direct LLM-Lasso was competitive in some experiments, while LLM-Assisted Regularization (AR) was strongest in others.Source: B. Part II Results · p. 8
  • More articles did not consistently improve accuracy: Direct LLM-Lasso improved for Mexico and Syria but not Ukraine, while larger corpora often harmed AR configurations.Standard Lasso improved for Mexico but performed worse for Syria and Ukraine.Source: Impact of number of articles · p. 9, Figure 4 · p. 10
  • Mexico’s gravity-plus-ratios model reached MAPE 17.15% (baseline 17.44%); Syria’s Direct LLM-Lasso reached 29.29% (control 31.44%); Ukraine’s AR reached 41.05% (gravity baseline 61.66%).Figure 3 shows APE distributions across forecasting periods in addition to these aggregate MAPE results.Source: Figure 3 · p. 9, Impact of number of articles · p. 9
  • Larger corpora often produced greater forecasting variance, particularly for Ukraine; observed differences are exploratory because each corridor has few monthly holdout forecasts.Smaller article configurations sometimes yielded narrower error distributions and lower median APE values; Direct LLM-Lasso generally showed wider distributions than AR.Source: Impact of number of articles · p. 9

VI. CONCLUSION

  • The framework uses news-derived signals indirectly through weighted Lasso regularization and push–pull feature construction; results emphasize article quality, contextual consistency, and stability alongside forecasting accuracy.The conclusion notes limitations associated with directly generated LLM-based regularization penalties.Source: §VI · pp. 9–10
  • Hierarchical inference pipelines extracted migration-related push–pull signals, and SuLuSa had the strongest overall classification behavior among the evaluated configurations.SuLuSa combined lightweight inference, larger-model reasoning, and full-text contextual analysis.Source: §VI · pp. 9–10

APPENDIX A URL-LEVEL CLASSIFICATION PROMPT

The URL-level prompt classifies migration pressure using URL tokens and interprets push and pull relative to the country named in the URL. It requires a constrained JSONL output with an English reason quoting triggering tokens.

APPENDIX A URL-LEVEL CLASSIFICATION PROMPT

  • Classification must use only lexical tokens in the URL, without outside knowledge or invented details.The reason must quote the exact triggering token or tokens.Source: Appendix A · p. 11
  • Push means conditions that may cause people to leave the URL-referenced country, while pull means conditions that may cause people to arrive.Reasons must be written in English even when URL tokens are non-English.Source: Appendix A · p. 11
  • The prompt assigns each URL one of five labels: push, pull, mixed, neutral, or unclear.Unclear applies when the URL is too short, generic, or ID-like.Source: Appendix A · p. 11
  • The required output is one JSONL object per row containing the row ID, pressure label, and a reason of at most 12 English words.Input consists of tab-separated row IDs and URLs.Source: Appendix A · p. 11

APPENDIX B LLM PROMPT FOR ARTICLE-BASED PUSH–PULL

The article-level prompt classifies migration pressure from article text using five labels and a country-relative interpretation of push and pull. It restricts evidence to the supplied article and requires a JSON response.

APPENDIX B LLM PROMPT FOR ARTICLE-BASED PUSH–PULL

  • The classifier must use only the provided article text and no outside knowledge or invented details.Its reason must be written in English.Source: Appendix B · p. 11
  • Push denotes conditions that may cause people to leave the main country discussed, while pull denotes conditions that may cause people to arrive.The prompt includes war, unemployment, and disasters as push examples, and jobs, visas, aid, and safety as pull examples.Source: Appendix B · p. 11
  • The article is classified as push, pull, mixed, neutral, or unclear based on migration-driving conditions in its text.Neutral means the article is not clearly about such conditions; unclear means its text is too vague.Source: Appendix B · p. 11
  • The prompt requires a JSON object whose pressure field contains one of the five labels.The output format is strict JSON with no additional text.Source: Appendix B · p. 11

APPENDIX C FEATURE-CATEGORY COUNTING PROMPT FOR LLM-ASSISTED REGULARIZATION

The feature-counting prompt asks an LLM to count distinct pieces of semantic evidence for migration-model features in a text source. It allows overlapping evidence to support multiple features rather than relying on keyword frequency.

APPENDIX C FEATURE-CATEGORY COUNTING PROMPT FOR LLM-ASSISTED REGULARIZATION

  • The prompt counts distinct pieces of evidence related to each feature using semantic meaning, not keyword frequency.It directs the model to use contextual meaning and overall article content.Source: Appendix C · p. 12
  • Features are not mutually exclusive, so one sentence may support multiple features and overlapping evidence may count in multiple categories.The prompt says not to force evidence into only one category.Source: Appendix C · p. 12
  • Counts are based on distinct claims, events, situations, or conditions, and unsupported features receive zero.Listed keywords are conceptual examples rather than terms to search mechanically.Source: Appendix C · p. 12
  • The prompt supplies contextual examples for feature categories including peace, human development, education, and economic conditions.The examples also encompass health, migration routes, and neighboring countries.Source: Appendix C · p. 12

APPENDIX D DIRECT LLM-LASSO PROMPT

The Direct LLM-Lasso prompt asks a language model to assign monthly feature-specific regularization penalties from article evidence. Lower weights indicate more important features that should receive less penalty, and the model is instructed to compare features rather than use mention counts.

APPENDIX D DIRECT LLM-LASSO PROMPT

  • Direct LLM-Lasso assigns monthly feature-specific penalties using article evidence for that month.The model estimates relative feature importance from contextual information in the corresponding article corpus.Source: Appendix D · p. 12
  • A lower weight means a feature is more important that month and should be penalized less, while a higher weight means stronger penalty.Weights are constrained to 0.1–1.0.Source: Appendix D · p. 12
  • The prompt requests one score for every listed feature and a JSON-only response.The input includes a feature block and article context for the month.Source: Appendix D · p. 13
  • The model must compare feature importance rather than assign identical penalties or use mention counts or inverse-count logic.Features need not be literally named in the articles to matter.Source: Appendix D · p. 13

APPENDIX E TEMPERATURE SENSITIVITY RESULTS

Classification distributions changed little across temperature settings, while SuLuSa remained the strongest inference hierarchy across repeated runs and evaluated temperatures.

APPENDIX E TEMPERATURE SENSITIVITY RESULTS

  • Qwen SuLuSa classified 163 articles as unclear at τ = 0.3 and 159 at τ = 1.5.Table IV reports category counts across evaluated inference pipelines and temperatures; distributions remained broadly stable.Source: Appendix E · p. 13, Table IV · p. 13
  • The constrained classification schema likely helped keep outputs consistent, while hierarchy configuration mattered more than temperature selection alone.The predefined categories limited unrestricted responses, and higher temperatures occasionally produced slightly more mixed classifications.Source: Appendix E · p. 13

APPENDIX F SULUSA ACROSS TEMPERATURES

  • SuLuSa consistently achieved the strongest overall performance across temperatures and repeated runs.The overall classification distribution remained broadly stable, despite minor variability at higher temperatures.Source: Appendix F · p. 13, Table V · p. 13

APPENDIX G ARTICLE RETRIEVAL QUERIES

The appendices describe corridor-specific, source-filtered news retrieval and report that the best forecasting configuration varied by corridor. They also document comparisons across article-corpus sizes and Lasso regularization strengths.

APPENDIX G ARTICLE RETRIEVAL QUERIES

  • Queries combined corridor-region and migration-event terms with source-domain filters, using generalized templates and corridor-specific terms.The terms included Ukraine/Russia with war-related keywords, Mexico/United States with migration-related keywords, and Syria/Turkey with conflict and displacement terms.Source: Appendix G · pp. 13–14

APPENDIX H BEST MODELS FOR COUNTRY PAIR

  • Article-corpus sizes refer to pre-constructed migration-news datasets loaded into forecasting experiments rather than generated during model training.For example, “Ukraine 100” and “Mexico 1245” identify experiments using CSV datasets with different article quantities.Source: Appendix H · p. 14
  • The strongest forecasting configuration varied by corridor: Mexico–United States favored No wj, Ukraine–Poland favored AR, and Syria–Turkey favored Direct LLM-Lasso.Table VI compares configurations across forecasting method, λ, and article corpus size.Source: Appendix H · p. 14, Table VI · p. 14

APPENDIX I LAMBDA VALUES

  • Multiple λ values were tested to compare weaker and stronger regularization under identical forecasting conditions across corridors.The selected values were used in later rolling forecasting experiments.Source: Appendix I · p. 14, Table VII · p. 14
Loading 2610.07208v1…