Source-linked AI summary
Can Large Language Models Anticipate Behavioral Responses to Social Policies? A Case of Pension Enrollment Prediction among China's Flexible Workers
Yumiao Li, Peixin Liu, Donglin Di, Chen Li, Runhuan Feng
TL;DR
Assessing behavioral responses to social-policy changes is difficult because econometric extrapolation can be unreliable and field pilots are costly. The paper develops FlexPension-LLM and DKI-RDistill to predict China’s hierarchical pension-enrollment decisions using policy-grounded cues and rationale supervision. It reports strong blind-test performance, cross-survey generalization, and expert-preferred decision traces.
Problem
Existing econometric and machine-learning approaches provide limited policy-tied explanations, while off-the-shelf LLMs may apply specialized institutional rules inconsistently in pension-enrollment prediction.
Method
FlexPension-LLM uses worker and policy information with Probit-derived indicators, while DKI-RDistill injects pension rules and distills rationale supervision with teacher-error regeneration and LoRA/SFT.
Results
0.9316 Composite F1 on the CHFS 2019 blind benchmark, outperforming 15 of 17 baselines, while averaging 0.7549 Composite F1 across four external surveys.
Takeaways & Limitations
Policy-grounded LLM adaptation delivers near-frontier predictive performance, stable cross-survey generalization, and structured decision traces for aggregate policy simulation and evaluation.
Takeaways & Limitations
The study is tied to China’s pension institutions, household-survey labels, and available policy parameters, and the model is not designed for individual advice or automated administrative decisions.
Abstract
from arXiv · showhide
Assessing the impacts of social policy changes is a widely acknowledged challenge for policymakers. Econometric methods can be unreliable when extrapolating to hypothetical scenarios, while field pilot programs are highly costly. In this paper, we propose using large language models (LLMs) as policy-assessment tools adapted from general-purpose models. We present FlexPension-LLM, the first domain-specialized large language model for a hierarchical pension-enrollment prediction task among flexible workers in China, and introduce DKI-RDistill, which injects policy-grounded cues into the prompt, including Probit-derived marginal effects and hukou-province pension rules. The method then uses LoRA/SFT to distill rationale-augmented supervision into an open-weight MoE student, with teacher errors corrected by regenerating those cases under ground-truth labels. On a CHFS 2019 blind split, FlexPension-LLM achieves 0.9316 Composite F1, surpassing its Claude Sonnet 4.5 teacher and 15 of 17 baselines, and is statistically indistinguishable from Claude Opus 4.6. Across four external surveys, it averages 0.7549 Composite F1 and shows the narrowest performance range among the strongest systems. Component analysis shows that gains come mainly from policy-grounded cue injection and error-filtered supervision, while rationales provide decision traces that can be checked against policy rules.
Introduction
The paper frames flexible workers’ pension enrollment in China as a policy-sensitive hierarchical decision and develops a domain-specialized LLM to predict it. FlexPension-LLM combines policy-grounded cues, rationale supervision, and error-filtered distillation, achieving strong blind-test and external-survey performance.
- Problem: Flexible workers first decide whether to participate, then choose between resident- and employee-pension channels under economic, household, historical, and institutional influences.Relevant cues include income, assets, prior participation, hukou-province rules, family pension status, access constraints, and contribution burden.
- Motivation: Standard econometric models capture population-level associations but are less suited to tracing how policy constraints and behavioral factors combine in individual profiles.Off-the-shelf LLMs may apply rules superficially and exhibit rational-actor bias, motivating domain adaptation.
- Approach: FlexPension-LLM takes worker, household, historical, policy, and Probit-derived indicators as inputs and outputs participation, pension channel, and a structured rationale.It is intended for aggregate prediction, policy simulation, and evaluating institutional effects on participation decisions.
- Approach: DKI-RDistill converts empirical priors and pension rules into model-readable cues, retains teacher-correct traces, and regenerates teacher-error cases under ground-truth labels before LoRA/SFT.The pipeline connects policy parameters with rationale-augmented supervision for an open-weight student model.
- Results: 0.9316 Composite F1 on the CHFS 2019 blind benchmark significantly outperforms 15 of 17 baselines, including the Claude Sonnet 4.5 teacher.Across four external datasets, the model averages 0.7549 Composite F1 and remains statistically comparable to Claude Opus 4.6 and Gemini 3.1 Pro Preview.
- Contributions: The paper contributes FlexPension-LLM, DKI-RDistill, and broad validation through blind testing, external datasets, ablations, robustness checks, boundary cases, and expert evaluation.The stated contribution is near-frontier prediction, cross-survey generalization, and structured traces for aggregate policy simulation.
Related Work
Prior work studies pension participation with econometric and machine-learning models, while newer LLM research explores behavior simulation but identifies reliability problems. This paper positions its approach at the intersection of policy-grounded reasoning, rationale distillation, and parameter-efficient specialization.
- Pension participation research: Econometric and machine-learning studies link pension participation to income, education, hukou status, age, family structure, compliance, enforcement, employment conditions, thresholds, frictions, benefits, and institutional access.These factors motivate the paper’s use of structured economic, household, historical, and institutional signals.
- LLMs as behavioral models: LLM studies suggest models can produce plausible behavior and population-conditioned responses, but outputs may be prompt-sensitive and inconsistent with domain-specific rules.The paper treats this reliability problem as central because pension participation depends on institutional constraints.
- Reliability and adaptation: Technical work improves LLM behavior through prompt engineering, knowledge injection, reasoning decomposition, self-consistency, tree-structured reasoning, and knowledge distillation.These methods provide the broader methodological context for policy-grounded adaptation.
- Reliability and adaptation: Structured reasoning and teacher-student distillation can transfer task-specific reasoning into smaller models, while rationale regeneration can align supervision with labels in difficult cases.The paper combines these ideas with econometric prior extraction and province-level pension-policy cues.
Task and Data
The study formulates pension enrollment as a two-layer institutional decision using structured worker, household, historical, and hukou-province policy information. CHFS 2019 supplies the main labeled dataset, while multiple external household surveys support transfer evaluation.
- Task formulation: The task predicts participation first and insurance channel second, producing three terminal labels: non-participation, resident pension insurance, and employee pension insurance.The hierarchy reflects the conditional structure of pension enrollment decisions.
- Intended use: Model outputs support aggregate counterfactual analysis of contribution thresholds, subsidies, and local rules across heterogeneous worker groups, not individual advice or administrative recommendations.The intended use is policy decision support at the aggregate level.
- Data: CHFS 2019 provides 15,672 flexible-worker samples: 8,842 participants and 6,830 non-participants.Participants comprise 56.4% of the constructed sample and non-participants 43.6%.
- Overview: Table 1 summarizes the task hierarchy, input modules, policy interface, and intended use.The table is the paper’s compact overview of task and data design.
- Data: External evaluation samples 500 label-stratified cases from each of CHFS 2017, CHIP 2018, CLDS 2018, and CFPS 2018.These surveys vary in organization, year, sampling frame, and questionnaire while preserving mappable pension labels and covariates.
Method
The method injects policy and econometric cues into teacher prompts, filters teacher errors through ground-truth regeneration, and adapts an open-weight student with rationale-augmented LoRA/SFT supervision.
- Domain Knowledge Injection: DKI-RDistill combines worker profiles, hukou-province rules, and Probit-derived priors into policy-grounded prompts for teacher generation.The pipeline separates cue construction from student adaptation, making institutional mechanisms explicit before distillation.
- Domain Knowledge Injection: DKI creates resident- and employee-pension burden ratios plus household pension dependency to expose affordability, family, and institutional cues.The burden ratios compare local minimum contributions with observed capacity signals, while dependency captures reliance on household pension transfers.
- Rationale-Augmented Supervision: Teacher-correct rationales are retained, whereas teacher-error cases are regenerated under ground-truth labels before forming the distilled supervision set.This construction preserves usable reasoning while redirecting incorrect terminal decisions into label-consistent rationale supervision.
- Student Adaptation: LoRA/SFT adapts an open-weight Qwen student to output both terminal labels and policy-grounded decision traces from corrected supervision.Base weights remain frozen while low-rank adapter parameters are trained on rationale-label target sequences.
Experimental Setup
The experiments use a held-out CHFS 2019 blind test, external out-of-domain datasets, and comparisons against prompted, fine-tuned, teacher, and closed-source baselines under hierarchical F1 metrics.
- Data Splits: From 15,672 CHFS 2019 samples, 2,350 are held out blindly, while 13,322 form the teacher-supervision pool.The distilled pool is split into 10,657 training and 2,665 validation samples.
- Evaluation Protocol: The blind test set is excluded from prompt tuning, supervision construction, fine-tuning, and checkpoint selection, while four external datasets are reserved for out-of-domain evaluation.The external datasets are never incorporated into student fine-tuning.
- Baselines: The benchmark compares FlexPension-LLM with Qwen-ZS, Claude Sonnet 4.5, Correct-only SFT, and recent closed-source baselines including Claude Opus 4.6.Claude Sonnet 4.5 serves as teacher, while Qwen 3.5-35B-A3B is the open-weight student base.
- Metrics: Composite F1 is the primary metric, computed as 0.6 × Action F1 + 0.4 × Type F1.Action F1 evaluates participation, while Type F1 evaluates employee-pension classification conditional on participation.
- Statistical Procedure: Paired differences use 95% stratified paired-bootstrap confidence intervals with B = 10,000 resamples and seed 42.The student is fine-tuned with LoRA/SFT and constrained to structured JSON outputs.
Results and Analysis
FlexPension-LLM performs near the frontier on blind and external benchmarks, while component and robustness analyses identify policy-grounded prompting and error-filtered rationale supervision as key contributors.
- Main benchmark: 0.9316 Composite F1 places FlexPension-LLM third among 18 blind-test models, with significant gains over 15 of 17 baselines.It exceeds the Claude Sonnet 4.5 teacher and remains statistically tied with the strongest systems.
- Cross-dataset generalization: 0.7549 external-average Composite F1 improves over Qwen-ZS by +0.1170 and over the Claude Sonnet 4.5 teacher by +0.0595.The external average summarizes four household-survey datasets.
- Cross-dataset generalization: FlexPension-LLM’s external Composite F1 range is 0.7471–0.7687, narrower than Claude Opus 4.6’s, while paired intervals among strongest systems cross zero.This supports competitive and stable cross-dataset generalization rather than a statistically decisive advantage over every strongest system.
- Component analysis: DKI raises Claude’s Composite F1 by +0.0668 on the CHFS 2019 all-sample evaluation and by +0.0623 on the blind subset.Prompt-only DKI applied directly to Qwen produces a small blind-subset gain with a confidence interval crossing zero.
- Component analysis: Rationale supervision adds complementary value: FlexPension-LLM is higher than Correct-only SFT by +0.0171 on the external average, although the paired confidence interval crosses zero.The rationales provide inspectable traces over affordability, household dependency, contribution history, and hukou-province access rules.
- Training efficiency and robustness: Full-data training reaches 0.9316 Composite F1, parse success remains 1.0 across temperatures 0.0, 0.2, 0.5, and 0.9, and checkpoint comparisons remain stable.Checkpoint 1800 has the highest point estimate at 0.9394, but its paired comparison with the final checkpoint crosses zero.
Conclusion and Limitations
The paper introduces FlexPension-LLM and DKI-RDistill for hierarchical pension-enrollment prediction, combining policy rules and empirical priors with rationale supervision. Its evaluation supports near-frontier performance, while deployment remains bounded by China-specific institutions, labels, and policy inputs.
- FlexPension-LLM is a domain-specialized model for hierarchical pension-enrollment prediction among flexible workers in China.
- Policy-grounded LLM adaptation delivers near-frontier predictive performance, stable cross-survey generalization, and expert-preferred decision traces for aggregate policy simulation and evaluation.
- The study remains tied to China’s pension institutions, household-survey labels, and available policy parameters, so transfer requires rebuilding policy inputs and validation data.
Ethical Statement
The model uses structured household-survey records and reports only aggregate, non-identifying results.
- FlexPension-LLM uses structured household-survey records and reports only aggregate, non-identifying results.
Code and Data Availability
Code and reproduction materials are publicly available, while the underlying household-survey microdata remain subject to their respective access conditions and are not redistributed.
- Code and reproduction materials are publicly available at the project repository.
- The underlying CHFS, CFPS, CHIP, and CLDS microdata are subject to their respective access conditions and are not redistributed.
Supplementary Material
The supplementary material documents the data flow, model selection, policy-cue construction, evaluation procedures, training diagnostics, and supporting examples. It complements the main claims with tables, figures, configuration details, and expert-evaluation summaries.
- Data split and evaluation sets: The supplement opens with the sample split and evaluation sets, establishing the data flow for subsequent tables.
- Data split and evaluation sets: CHFS 2019 flexible-worker samples are divided into train–validation and isolated blind-test splits, with four external surveys contributing 500 sampled cases each.
- Model selection and benchmarking: Table S1 combines candidate screening for teacher selection with the full blind-test Composite F1 ranking.
- Prompt and policy-cue construction: The DKI Prompt adds household, affordability, and policy-rule cues while retaining the same structured JSON target.
- Prompt and policy-cue construction: Policy parameters are matched to each case by hukou province, with Beijing shown as an example field preview.
- Supporting analyses: The supplement documents affordability distributions, Probit estimates, descriptive statistics, external ablations, boundary cases, and blind-test error profiles.
- Expert evaluation: The expert evaluation used 27 respondents who assessed 12 cases spanning all-correct, model-specific, disagreement, and all-wrong strata.
- Evaluation metrics: Composite F1 is defined as 0.6× Action F1 + 0.4× Type F1, with blind ranks based on point estimates and paired-bootstrap uncertainty reported separately.