Source-linked AI summary
From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring
Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan, Arash Hajikhani
TL;DR
Cross-country STI policy surveys are costly and difficult to scale consistently. This paper uses an LLM pipeline to generate structured survey responses from policy texts and compares them with human responses. It reports high overlap for structured indicators, with differences in free-text responses, and points to hybrid workflows with continued human oversight.
Problem
Cross-country STI policy surveys rely on expert reporting and face scale, consistency, and timeliness challenges as policy initiatives expand.
Source: §1 · pp. 1–2Method
The paper applies long-context prompting to map policy texts into STIP Compass survey fields, adds LLM validation, and compares generated responses with human responses.
Source: §1 · p. 2Results
LLM–human overlap was 84–95% for structured indicators; free-text responses diverged, with models tending toward procedural detail and humans toward contextual interpretation.
Source: §5.5 · p. 13Takeaways & Limitations
LLMs can pre-fill structured surveys while human experts refine, contextualize, and interpret responses in a hybrid workflow.
Source: §5.5 · p. 13, §6 · p. 14Policy initiatives are heterogeneous and institutionally embedded, and relevant context is not always captured in publicly available sources.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Science, technology, and innovation policies are crucial for competitiveness, yet their diversity and scale make them difficult to map and monitor consistently. Existing approaches rely heavily on manual survey efforts, which are costly and challenging to scale across countries. Large language models (LLMs) enable new possibilities for extracting and structuring information from long and unstructured policy documents. This paper presents an application of LLMs as "AI respondents" for generating structured survey responses from policy texts. We develop a data extraction pipeline based on long-context in-context learning to map information from public web sources into predefined survey categories, including policy instruments, target groups, and thematic areas. The pipeline integrates a validation step using a secondary LLM to assess relevance and evidence, alongside comparisons with human-provided responses. Using a multi-country dataset, we evaluate the alignment between LLM-generated and human-generated outputs through overlap measures and cross-validation. Results show that LLMs achieve high agreement for structured indicators (84-95%), while differences remain in free-text fields, where models tend to provide more detailed procedural descriptions. These findings highlight the potential of hybrid human-AI workflows for policy monitoring, improving both efficiency and scalability while maintaining the need for human validation and contextual interpretation.
1 Introduction
International STI policy monitoring depends on costly expert surveys that are difficult to scale consistently as initiatives grow in number and complexity. The paper proposes an LLM pipeline for filling survey categories from policy texts and reports high overlap with human responses for structured codes, alongside differences in free-text fields.
1 Introduction
- STI policies are complex, and expert-led international surveys face scale, consistency, and timeliness challenges as initiatives expand.The STIP Compass relies on expert respondents to complete structured surveys linked to web-based evidence.Source: §1 · pp. 1–2
- Policy initiatives are heterogeneous and institutionally embedded, and those contexts are not always captured in public sources.The paper also notes risks of bias and distortion from repeatedly training on synthetic outputs.Source: §1 · p. 2
- 84–95% overlap between LLM and human responses was reported for structured policy codes, while LLM text was more procedural and human text more contextual.The pilot covered six OECD countries.Source: §1 · p. 2
- The pipeline uses long-context prompting to map policy instruments, target groups, and themes from web-scraped content into survey responses.It also generates free-text descriptions and objectives.Source: §1 · p. 2
- A secondary LLM validation layer assesses response relevance and evidence against the source material.The paper compares outputs with human responses and explores complementarities in descriptive and objective fields.Source: §1 · pp. 2–3
2 Related Work
Prior work identifies persistent costs and inconsistencies in compiling cross-country STI policy data, while LLMs offer adaptable information extraction and survey-response generation. This paper tests an operational pipeline against human-provided responses across multiple dimensions of policy data.
2 Related Work
- Large-scale cross-country policy data are costly to compile and validate, while expert-driven approaches face reporting burden, data gaps, and national-context inconsistencies.Source: §2 · pp. 3–4
- LLMs have been used for extraction, summarization, and question answering [24], and in-context learning adapts to survey questions without costly labeled training data.Prior studies also suggest LLMs can act as proxies for human subjects in social-science experiments.Source: §2 · p. 4
- Recursive training on synthetic data raises concerns about distributional drift and output degradation, while heterogeneous initiatives and limited gold-standard labels constrain fine-tuning.Source: §2 · p. 4
- The paper's contribution is comparing human-provided and AI-generated STI policy responses across multiple data dimensions within an operational STIP Compass pipeline.It situates this work among calls to use the “new data frontier” in innovation studies.Source: §2 · p. 4
3 Methodology
The methodology processes OECD-sourced initiative texts, applies long-context prompting to fill survey fields, validates generated responses, and evaluates their alignment with human responses and fine-tuned models.
3.1 Data Preparation
- Initiatives under 200 tokens were discarded, while texts exceeding 120,000 tokens were summarized in 50,000-token chunks and then aggregated.The chunk prompts requested policy descriptions, objectives, dates, themes, instruments, stakeholders, budgets, and evaluations, using only source text and summaries.Source: §3.1 · pp. 4–5
3 Methodology
- The workflow moves from OECD-scraped policy texts through preparation and survey-prompt design to GPT-4o-128k extraction, followed by human comparison and fine-tuned-model cross-validation.Source: §3 · p. 4, Figure 1 · p. 5
3.2 Pre-filling - Long-Context In-Context Learning
- Long-context in-context learning places full content, survey questions, and extraction guidelines in one extended prompt to preserve coherence and capture nuanced findings.The approach was chosen over RAG because extraction knowledge was not explicitly defined and split-question prompts could create redundancy; the context window fit most dataset cases.Source: §3.2 · p. 6
Prompt Design
- Prompts incorporated OECD guidelines and requested structured English responses for descriptions, objectives, policy labels, dates, budgets, and evaluation reports.Source: Prompt Design · p. 6
Response Validation
- A second LLM scored each response for evidence support and relevance to the prompt and source material, filtering cases that could reflect hallucinations or misinterpretations.Source: Response Validation · p. 6
3.3 Evaluations
- Evaluation combined human–LLM overlap, label-wise agreement categories, and k-fold cross-validation with fine-tuned masked and causal models.Cross-validation addressed the lack of a full gold-standard reference for all structured policy indicators.Source: §3.3 · pp. 6–7
4 Experimental Setup
The setup tested whether web-scraped content could support survey pre-filling and map unstructured information to structured categories. It used GPT-4o-128k, LLM-based validation and comparison, and a six-country cost assessment.
4 Experimental Setup
- The study asked whether web-scraped content could pre-fill STIP Compass questions and map unstructured information into structured survey categories.It hypothesized that long-context LLMs could capture structured information in free-text fields and identify multi-label policies.Source: §4 · p. 7
4.1 Choice of LLM
- 0.938 was GPT-4o’s mean win rate on standard evaluation metrics, supporting its selection for the study.The authors selected GPT-4o-128k for its extended context window and benchmark performance.Source: §4.1 · p. 7
4.2 Evaluation Design
- A second GPT-4o-128k instance checked generated responses against prompts and source material, while post-extraction evaluation compared human and LLM free-text responses.The comparison categorized overlap into four levels; policy-label agreement and five-fold cross-validation were evaluated using micro F1 scores.Source: §4.2 · p. 7
4.3 Implementation Details
- Azure AI Services supported preparation and evaluation, and each cross-validation fold used an 80%/10%/10% train-validation-test split.Dataset analysis and agreement scores were computed with standard Python libraries.Source: §4.3 · p. 8
4.4 Cost
- €446.84 was the GPT-4o cost for the six-country pilot, not a full OECD deployment.The authors note that full-scale costs would rise with document volume and length, and list URL filtering, cached scraping, selective summarization, and targeted human review as ways to reduce them.Source: §4.4 · p. 8
5 Results & Discussion
LLM-generated responses aligned strongly with human-provided structured indicators, while free-text responses often differed in overlap and emphasis. The findings support a hybrid workflow in which LLMs pre-fill surveys and human experts validate and contextualize responses.
5.1 Dataset Analysis
- Table 2 distinguishes initiatives with missing or short web content, sufficient content without identifiable policy instruments, and suitable content, and reports usable sample counts by country.Figure 2 showed similar expert and LLM label distributions, but both were imbalanced, with many labels occurring infrequently.Source: §5.1 · pp. 8–9, Table 2 · p. 9, Figure 2 · p. 9
5.2 Human-LLM Overlap
- Descriptions had high overlap in 74.05% of cases, while full overlap was 1.19%.For objectives, high overlap was 41%, no overlap about 36%, low overlap 22%, and full overlap 1%; the authors associate differences with variation in approach, detail, scope, and available information.Source: §5.2 · p. 10, Figure 3 · p. 10
- The LLM identified at least one human-provided target group in 95% of initiatives on average, versus 84% for themes and 85% for instruments.These are average overlap rates; the authors report that country-level variation was generally limited.Source: §5.2 · pp. 10–11, Table 4 · p. 11
5.3 Human-LLM Agreement
- Clear, unambiguous policy-indicator definitions aligned with higher agreement, while abstract or broad labels, category overlap, and general terminology aligned with lower agreement.Agreement scores were dispersed across labels and dimensions.Source: §5.3 · pp. 11–12, Figure 4 · p. 11
5.4 Cross-Validation
- Causal models captured longer documents more strongly and achieved higher multi-label F-scores than masked models; BigBird variants outperformed RoBERTa.The authors attribute the differences to context length and architecture, including BigBird’s block-sparse attention mechanism.Source: §5.4 · p. 12, Table 5 · p. 12
5.5 Discussion
- LLMs tended to provide more procedural detail, while human respondents emphasized contextual and societal implications of policy initiatives.The authors describe these differences as complementarity rather than simple substitution, and recommend that experts validate objectives and adjust contextual framing.Source: §5.5 · pp. 12–13
- The authors identify risks from over-reliance on synthetic outputs, including bias, redundancy, and model collapse, and call for human oversight and source triangulation.They identify free-text fields as especially important for safeguards because hallucination and framing bias are more likely there than in constrained taxonomy selection.Source: §5.5 · p. 13
6 Conclusion
The integrated STIP Compass pipeline complements human expertise by achieving high overlap with human responses on structured indicators while adding procedural detail to free-text fields. The authors highlight efficiency and scalability benefits alongside the need for oversight and broader evaluation.
6 Conclusion
- The pipeline achieves high overlap with human-generated responses across structured indicators while providing additional procedural detail in free-text fields.The authors describe the approach as complementing human expertise in international surveys.Source: §6 · p. 13
- LLMs can pre-fill structured categories, while human respondents contribute contextualization and societal framing to policy descriptions.The authors identify careful oversight as necessary to address risks of bias, redundancy, and over-reliance on synthetic outputs.Source: §6 · p. 14
- Future work should extend the study beyond six pilot countries, refine validation, and examine how to embed human-AI collaboration in STI policy monitoring.The conclusion frames the approach as a step toward more scalable, consistent, and timely global policy intelligence.Source: §6 · p. 14
A Hyperparameters
Tables 6 and 7 are identified as reporting hyperparameters for masked encoder and causal models, respectively.
A Hyperparameters
- Table 6 reports hyperparameters for masked (encoder) models.The supplied passage gives the table’s scope but no parameter values.Source: Table 6 · p. 17
- Together, Tables 6 and 7 cover masked encoder and causal model hyperparameters.The captions distinguish the model types covered.Source: Table 6 · p. 17, Table 7 · p. 17
- Table 7 reports hyperparameters for causal models.The supplied passage gives the table’s scope but no parameter values.Source: Table 7 · p. 17
B Prompts
The appendix prompts specify binary validation of evidence and relevance, a categorical comparison of free-text responses, and structured extraction prompts for survey fields. They prescribe output formats and instructions for policy categories and initiative details.
B.1 Validation Prompt
- The validation prompt separately scores whether a response is evidenced and relevant, using 0/1 assessments returned as JSON.The evidence judgment concerns support in the text; relevance concerns following the instructions.Source: §B.1 · pp. 17–18
B.2 Free-Text Evaluation Prompt
- The free-text evaluation prompt asks for similarities, overlaps, and differences between human and LLM assessments, selecting one of four overlap categories.Its examples illustrate Full, High, Low, and No overlap with an observation.Source: §B.2 · pp. 18–19
B.3 Pre-Filling Prompts
- Separate prompts request concise English descriptions and objectives grounded in the source text, with responses formatted as JSON.Descriptions and objective explanations are each capped at 100 words.Source: §B.3 · pp. 19–20
- The start-date prompt asks for the initiative’s starting date when mentioned and formats the date with an explanation in JSON.Its examples allow year-and-month dates and an n.a. response when no start date is mentioned.Source: §B.3 · p. 20
- The pre-filling prompts classify policy instruments, target groups, and themes against supplied labels or definitions, returning relevant matches with short justifications.The prompts specify structured JSON-array responses for these category fields.Source: §B.3 · pp. 20, 22, 23, 24
- The policy-theme prompt supplies theme codes, labels, and defining questions to guide selection of relevant themes.The prompt requests a short English justification and specifies an n.a. response when no theme is relevant.Source: §B.3 · pp. 23–26
- Other prompts extract budget or expenditure details and evaluation information, using “No information” or n.a.-style outputs when absent.The evaluation prompt requests the evaluation name and information about the evaluation.Source: §B.3 · pp. 26–27
B.4 System Prompt for Fine-tuning Causal Models
- A separate system prompt for fine-tuning causal models requests only a flat list of clearly relevant labels across the three policy dimensions.It names Policy Instruments, Policy Target Groups, and Policy Themes.Source: §B.4 · pp. 27–28