PaperlayerView exact revision on arXiv

Source-linked AI summary

GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

Veronika Solopova, Ivan Havlytskyi, Viktoria Skorik, Ostap Vykhopen

arXiv:2610.12617v1cs.CLcs.IR

TL;DR

Geopolitical intelligence reports are heterogeneous and analytically complex, leaving the reliability of grounded QA over them unclear. GeopoliticalQA builds a document-aware benchmark and evaluates retrieval and generation; retrieval patterns vary by level, while answer generation remains difficult even with gold evidence.

  • Problem

    The reliability of grounded QA over heterogeneous reports that discuss uncertainty, competing narratives, and future risks remains unclear.

    Source: §1 · p. 1

  • Method

    GeopoliticalQA profiles reports before generating grounded questions, validates dataset quality with expert annotation and an LLM judge, and evaluates retrieval and answer generation.

    Source: §1 · pp. 1–2, §3.4 · p. 4

  • Results

    Document-level retrieval was similar across systems, chunk-level quality did not track embedding scale, and flawed answers rose from 28% for factual to 70% for risk-oriented questions despite gold evidence.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The benchmark shows that identifying relevant material can be achievable while selecting the right chunk and faithfully answering inference, comparison, and risk questions remain challenging.

    Source: §6 · p. 8

    The English-language corpus is strongly contemporary, with 94% of documents published since 2022, and is not representative of geopolitical analysis globally.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for retrieval-augmented question answering. We introduce a benchmark for grounded question answering over 415 open-source geopolitical and strategic intelligence reports, comprising 5,525 document-grounded question-answer pairs spanning factual, inferential, comparative, and risk-oriented questions. We propose a document-aware generation pipeline that profiles reports before generating questions and validate dataset quality through expert annotation and an LLM-as-judge protocol. Using the benchmark, we evaluate 15 retrieval systems and six open-weight LLMs for RAG. We find that models perform similarly on document-level retrieval, while at the harder chunk level retrieval quality is not driven by embedding-model scale: bge-m3, a mid-size open encoder matches an order of magnitude larger ones and outperforms commercial embedding APIs. In contrast, answer generation remains challenging even given the gold evidence, particularly for inferential and risk-oriented questions.

1 Introduction

Geopolitical intelligence reports hold substantial expert knowledge, but their heterogeneity and emphasis on uncertainty make reliable grounded question answering difficult to assess. GeopoliticalQA addresses this gap with a document-aware benchmark and evaluates scalable approaches to its construction.

1 Introduction

  • Geopolitical and strategic reports combine expert knowledge across security and policy issues, offering a growing resource for analysts.Potential uses include strategic communications, intelligence analysis, crisis response, policy development, and corporate risk assessment.Source: §1 · p. 1
  • Heterogeneous report types and discussions of uncertainty, competing narratives, and future risks make grounded QA difficult to evaluate with conventional question-generation strategies.The authors state that, to their knowledge, no benchmark systematically evaluates QA over these documents.Source: §1 · p. 1
  • GeopoliticalQA introduces a document-aware pipeline that profiles what each report can answer before generating grounded, document-appropriate question-answer pairs.The benchmark covers 415 reports and 5,525 pairs across diverse document types.Source: §1 · pp. 1–2
  • The paper also tests whether LLM judges can reduce expert-annotation costs and whether small open-weight models can support reproducible, cost-effective dataset construction.Source: §1 · p. 2

2 Related Work

Prior work shows that question answering over long documents and domain-specific report collections is challenging and merits grounded evaluation. Geopolitical intelligence reports add heterogeneous analysis, uncertainty, forecasts, and recommendations to this setting.

2 Related Work

  • Narrative and long-context QA benchmarks study evidence retrieval and integration across lengthy documents, including diverse tasks and document lengths.Source: §2 · p. 2
  • FinanceBench, ESG-report QA, and enterprise Q&A benchmarks assess RAG on specialized collections, but not geopolitical intelligence reports.The cited studies include FinanceBench (Islam et al., 2023), Parikh and Penfield (2024), and Filice et al. (2025).Source: §2 · p. 2
  • Geopolitical QA differs through source, language, and perspective variation, while intelligence reports combine facts with interpretation, uncertainty, forecasts, and recommendations.Li et al. (2025) focus on fact-seeking questions over multilingual Wikipedia passages, while this paper addresses heterogeneous long-form expert analysis.Source: §2 · p. 2

3 Methods

The methods build a diverse, English-language report corpus, index it for hybrid retrieval, and generate document-grounded questions through profiling and content gating. Human annotations validate an LLM judge, which scores the resulting benchmark and defines quality tiers.

3.1 Corpus Collection

  • The corpus comprises 415 open-source reports from 122 publishers, with a median publication year of 2025 and 94% dated 2022 or later.Document types deliberately range from analytical assessments to trackers, annual reports, policy briefs, and codebooks.Source: §3.1 · pp. 2–3
  • A structured source-selection prompt sought variation across publisher type, geography, epistemic stance, genre, and time; collection used disjoint publisher assignments and preventive English-language and access screening.Co-publications and near-duplicate document reissues were not tested; publisher partitions make the corpus a publisher taxonomy rather than a topic taxonomy.Source: §3.1 · p. 3

3.2 Parsing, Chunking, and Indexing

  • Documents were converted to structured Markdown and divided into hierarchical, section-aware chunks; the 55,353 chunks were embedded with multilingual-e5-large-instruct and stored in Milvus.Hybrid retrieval fuses dense cosine similarity and BM25 with Reciprocal Rank Fusion, returning the top eight chunks.Source: §3.2 · pp. 3–4

3.3 Q&A Generation

  • Per-document profiling uses eight semantic probes to build a representative chunk set and a profile specifying document type, purpose, and answerable or unanswerable scopes.Generation selects only supported analyst personas, emits atomic questions with grounding pointers, and applies constraints on forecasts, comparisons, and reported claims.Source: §3.3 · p. 4
  • For answering, each question is re-retrieved with hybrid top-eight retrieval, and the model must answer from those chunks or return INSUFFICIENT_CONTEXT.Answers include chunk citations, with modal hedging for inferential and risk-oriented questions.Source: §3.3 · p. 4
  • 5,525 question-answer pairs were generated from 413 of the 415 collected reports, averaging 13.4 questions per document.Two reports were skipped because they yielded no retrievable chunks.Source: §3.3 · p. 4

3.4 Validation

  • Quality is rated on six 1–3 dimensions, with three analyst annotators rating 250 distinct items; mean quadratic-weighted κ was 0.24 despite high raw agreement.The overlap block comprised 100 items, and annotators mainly differed at the good-versus-acceptable boundary.Source: §3.4 · pp. 4–5, Table 3 · p. 5
  • GPT-5.4 was adopted as the automatic judge because it matched the two consensus annotators at 80% and 75% exact agreement, within the reported human inter-annotator range.On 250 annotated items, low-effort scoring was statistically equivalent to default scoring, with identical exact agreement and macro-F1 with humans and an unchanged overall mean of 2.5.Source: §3.4 · p. 5

4 Results

The results cover 5,525 generated pairs and assess dataset quality, retrieval, and reader generation. Retrieval leaders include a mid-size encoder, while answer quality remains lower on harder questions.

4 Results

  • Of 5,525 question-answer pairs, 4,980 meet the cleaned-tier threshold and 3,503 meet the stricter gold-tier threshold.The primary tier requires every dimension to be rated at least 2; the gold tier requires every dimension to be rated 3.Source: §4 · p. 5

4.1 Dataset Quality

  • Dataset quality averages 2.60/3, with 66% rated excellent; faithfulness is weakest at 2.60 and has the only non-trivial broken rate (8%).Document-purpose fit, clarity, and atomicity average 2.99, 2.97, and 2.92, respectively.Source: §4.1 · p. 5

4.2 Retrieval Benchmark

  • bge-m3 (568M) scores 0.785 chunk nDCG@10, matching Qwen3-Embedding-8B (0.784) and statistically indistinguishable from BM25 (0.787), while significantly outperforming commercial APIs.Dense retrieval generally scales with encoder size, but not universally: gte-base significantly exceeds gte-large.Source: Figure 1 · p. 6, §4.2 · p. 6

4.3 RAG Reader Benchmark

  • Mid-tier readers outperform small readers in mean overall score (2.53 vs. 2.35); Llama-3.3-70B (2.56) and Mistral-Small-3.2 (2.55) are not statistically different.Qwen3-8B has the highest correctness score (2.66), an exploratory dimension.Source: §4.3 · p. 6

5 Error Analysis

Retrieval usually finds relevant material, but completeness declines for harder question types. Even with gold context, answer flaws rise substantially from factual to risk-oriented questions.

5 Error Analysis

  • BM25 retrieves an answer-cited chunk in the top 10 for 93–96% of queries and the source document for about 97%, while cited-set recall ranges from 89% on factual to 78% on risk-oriented queries.The chunk nDCG@10 spread reflects completeness—recovering the average 1.55 cited chunks—rather than a failure to find anything relevant; residual misses are not type-driven.Source: §5 · pp. 6–7
  • With gold context, flawed answers rise from 28% of factual items to 38% comparative, 47% inferential, and 70% risk-oriented items.These rates include minor issues as well as broken answers; score-1 failures alone do not account for the full flawed share.Source: §5 · p. 7
  • Both reader tiers show declining answer quality from factual to inferential to risk-oriented questions, with the small–mid gap widest on harder reasoning-heavy types.The figure orders comparative after inferential, despite comparative items having a lower flawed rate.Source: Figure 2 · p. 7, §5 · p. 7
  • Among score-1 failures, hallucination accounts for 9%, incompleteness or incorrectness for 7%, and unjustified refusal for 2%; smaller readers refuse more often.Mistral-Nemo refuses on 18% and Llama-3.1-8B on 14%, mostly unjustifiably; mid-tier readers refuse at most 5%, and Qwen models at most 1%.Source: §5 · p. 7

6 Discussion and Conclusion

Across 415 reports and 5,525 Q&A pairs, relevant source material is often retrievable, but finding complete supporting evidence and generating faithful answers remain difficult. The challenge is especially pronounced for questions requiring inference, comparison, or risk assessment.

6 Discussion and Conclusion

  • Document-level retrieval was similar across evaluated systems, while chunk-level quality was not driven by embedding scale; bge-m3 (568M) matched a model an order of magnitude larger and outperformed commercial embedding APIs.BM25 was also competitive at chunk level, but was reported for reference because it contributed to construction of the gold chunks.Source: §6 · p. 8
  • 28% of factual answers were flawed, compared with 38% of comparative, 47% of inferential, and 70% of risk-oriented answers, even when LLMs received gold evidence.Errors included unsupported claims, incomplete or incorrect answers, and refusals despite sufficient evidence.Source: §6 · p. 8
  • The selected LLM judge performed within the range of human agreement, while modest inter-annotator agreement (κ = 0.24) underscored the continued importance of human validation.The authors identify subjective, cognitively demanding judgments as a reason validation remains important.Source: §6 · p. 8
  • Retrieving relevant material is often achievable, but selecting the right chunks and faithfully answering from evidence remain open challenges, especially for inferential, comparative, and risk-oriented questions.The authors suggest improving multi-passage completeness through hybrid retrieval, reranking, query decomposition, or evidence aggregation, and controlling attribution and uncertainty in generation.Source: §6 · p. 8

Limitations

The benchmark’s scope and evaluation constrain how broadly its findings can be interpreted. Its contemporary, English-language corpus, model-mediated question distribution, and limited annotation and evaluation design leave important settings untested.

Limitations

  • The English-language corpus is contemporary (94% of documents were published since 2022) and shaped by source-selection and accessibility criteria, so it is not representative of global geopolitical analysis.These boundaries limit conclusions about multilingual analysis and historical collections.Source: Limitations · p. 8
  • Question generation and answering are model-mediated, and the question-type distribution—particularly the predominance of factual questions—reflects document affordances and the generation pipeline.The human-annotated sample is relatively small, agreement is modest, and the annotators are the authors and colleagues who helped design the pipeline, potentially biasing judgments upward.Source: Limitations · pp. 8–9
  • The evaluation is single-run, scores each answer with one judge, defines gold using chunks cited by an LLM-written reference, and does not study reranking, iterative retrieval, or long-context readers.Bootstrap confidence intervals cover items, not decoding or judge stochasticity; deduplication credit and by-type recall mitigate but do not eliminate citation bias.Source: Limitations · p. 9

Ethics Statement

The benchmark uses public reports as research material, not verified ground truth, and cautions against operational use without human oversight. The authors describe the data’s provenance, annotation, and model-generation safeguards.

Ethics Statement

  • The corpus consists of publicly available reports from their originating organisations; the paper provides publisher attribution and original links, shares the collection on request, and will remove items at rights holders’ requests.The documents themselves are not republished.Source: Ethics Statement · p. 9
  • The reports reproduce their sources’ framing and potential biases and should be treated as research source material, not verified ground truth; deployment in operational decision-making warrants human oversight.The caution reflects that LLMs can produce fluent but unsupported claims.Source: Ethics Statement · p. 9
  • No private individuals’ personal data or human subjects were involved; authors and colleagues annotated professionally with informed consent and no study-participant compensation.A self-hosted open model was used after commercial API filters refused some geopolitical material, avoiding a dataset biased toward filter-permissible content.Source: Ethics Statement · p. 9

A Data Availability

This appendix section describes the released research artifacts and limits on redistributing source documents, and notes possible pretraining contamination. It also documents a Stage-1 prompt for inventorying candidate report sources and prioritizing collection to build a geographically and institutionally varied corpus.

A Data Availability

  • The project releases derived chunks, 5,525 Q&A pairs and metadata, gold chunks, judge scores, and retrieval and reader outputs under CC-BY-4.0, but not source PDFs.Each item retains publisher attribution and an authoritative link; researchers may request the full collection.Source: A Data Availability · p. 10
  • Benchmark results may reflect pretraining exposure to portions of the public documents, so they measure retrieval and grounded synthesis rather than closed-book recall.Source: A Data Availability · p. 10

B Stage-1 Source-Selection Prompt

  • The source-selection methodology is a reproduced prompt that prioritizes regional and local think tanks for dataset diversity and seeks broad coverage across regions and source types.It specifies English-language documents, actively seeks non-Western institutional origins, and aims to maximize variance in institutional type, geographic origin, epistemic stance, genre, and temporal coverage.Source: B Stage-1 Source-Selection Prompt · pp. 10–11
  • The prompt requests source profiles covering institutional identity, focus, publication series, formats, cadence, access, archive depth, epistemic stance, variance, bias, corpus path, and priority.It also asks for sources to be grouped into categories such as official and multilateral, think tanks, humanitarian and rights, specialized domains, and academic outlets.Source: B Stage-1 Source-Selection Prompt · p. 12
  • The prompt requests a final gap analysis, collection sequencing recommendation, and practical bias-mitigation checklist.Sequencing should consider source yield, access model, and variance contribution.Source: B Stage-1 Source-Selection Prompt · p. 12
  • The prompt favors predictable publication schedules, stable URLs, named series, open or registration-only access, archives of at least three years, and downloadable files.It says to flag mixed access models and not to exclude sources solely for an advocacy stance, treating bias diversity as a corpus value.Source: B Stage-1 Source-Selection Prompt · p. 12

C Question-Type and Persona Distribution

The section reports question-type and analyst-persona distributions, with persona coverage reflecting which personas document profiling found supportable.

C Question-Type and Persona Distribution

D Example Q&A Pairs

This appendix presents representative question–answer pairs alongside the prompts used to judge benchmark items and generate document-grounded questions. The generation prompts prioritize answerability and grounding across question types, with stricter qualification and attribution requirements for inferential and risk-oriented answers.

D Example Q&A Pairs

  • Table 10 presents one representative item per question type, with bracketed markers pointing to retrieved source-report chunks that form the answer-cited gold.These citations refer to the retrieved-chunk set, not bibliography references.Source: D Example Q&A Pairs · p. 12, Table 10 · p. 13

E Judge Prompt

  • The item judge scores questions and answers against retrieved chunks, while a separate reader judge evaluates answers against gold context and uses the reference answer only for correctness.The item judge uses six dimensions; the reader judge uses three.Source: E Judge Prompt · pp. 12–13

F Generation Prompts

  • Generation prompts profile what documents can and cannot answer, then require atomic, excerpt-supported questions while excluding unsupported predictions and comparisons.They also direct the generator to match questions to document purpose and vary difficulty with supported inferential, within-document comparative, and risk questions.Source: F Generation Prompts · p. 14
  • The question-generation task requests up to a capped number of distinct, excerpt-answerable questions, preferring fewer well-grounded questions and spanning factual, inferential, comparative, and risk-oriented types where possible.The prompt discourages defaulting to factual recall alone.Source: F Generation Prompts · p. 15
  • Questions are returned as JSON with an atomic, self-contained question, analyst persona, type, rationale, and supporting excerpt indices.The analyst lens affects tone and emphasis, and the supplied excerpts are provided as numbered chunks.Source: F Generation Prompts · p. 15
  • Answer instructions require responding strictly from supplied excerpts, returning exactly 'INSUFFICIENT_CONTEXT' when unsupported, citing only chunks that support each claim, and avoiding outside knowledge.Answers must not assume omitted facts.Source: F Generation Prompts · p. 15
  • The prompts restrict named entities to those explicit in cited excerpts and require modal qualifiers for inferential or risk claims without attributing unsupported positions to actors.The specified qualifiers include “may,” “could,” and “might.”Source: F Generation Prompts · p. 15

G Embedding Query/Passage Prefixes

Embedding families use their recommended instruction prefixes, while BM25 uses none; Table 11 gives the full Qwen3 query instruction.

G Embedding Query/Passage Prefixes

  • Each embedding family uses its recommended instruction prefix, while BM25 uses none.Source: G Embedding Query/Passage Prefixes · p. 15
  • Table 11 lists the query and passage prefixes used for each embedding family.Source: Table 11 · p. 15
  • Table 11 gives the full Qwen3 query instruction: “Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: ”.Source: Table 11 · p. 15
Loading 2610.12617v1…