PaperlayerView exact revision on arXiv

Source-linked AI summary

CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information

Dingying Liu, Yunshun Zhong, Wentao Zhang, Yiyuan Li

arXiv:2609.08094v1cs.AI

TL;DR

Evaluating civic information search agents is challenging because authoritative answers vary across countries and jurisdictions, while realistic assistants use web search. CIVI evaluates agents across those contexts and uses ARISE to diagnose failures; 72.1% were retrieval-bound, and none matched the attentive human baseline.

  • Problem

    Existing government-information benchmarks are limited to a single country, despite authoritative answers varying across countries and jurisdictions and realistic assistants using web search.

    Source: §1 · p. 1

  • Method

    CIVI spans countries, jurisdictional levels, and United Nations functional categories, while ARISE combines source-injection ablation and search-trace analysis to classify failures into four modes.

    Source: §2.1 · p. 2, §1 · p. 2

  • Results

    72.1% of observed failures were attributed to retrieval-bound causes, and none of the ten evaluated agents matched the attentive human baseline.

    Source: §4.1 · p. 5, §7 · p. 9

  • Takeaways & Limitations

    Failure localization indicates whether interventions should target search invocation, authoritative retrieval, answer-evidence consistency, or comprehension support and human escalation.

    Source: §7 · p. 9

    The evaluation covers only English-language civic information in Canada, the United States, and Australia, limiting its geographic and linguistic scope.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models are increasingly deployed in public-sector settings, where incorrect guidance can cause irreversible harm. We introduce CIVI, the first framework for diagnosing search agent failures in civic information. Its benchmark instantiation jointly spans cross-national, interjurisdictional government contexts (federal, state, and local) and functional categories from an internationally adopted United Nations standard. We evaluate ten frontier search agents and find that none matches an attentive human baseline. Alongside accuracy, CIVI measures search invocation rate, selective no-search accuracy, and how often agents cite authoritative government sources. To perform this diagnosis, we introduce ARISE, which decomposes agentic search failures into four mutually exclusive modes, isolated via source-injection ablation. ARISE attributes 72.1% of all observed failures to retrieval-bound causes rather than to gaps in the models' parametric knowledge.

1 Introduction

CIVI evaluates agentic search for authoritative civic information across countries, government levels, and functions, while diagnosing why search agents fail. Its benchmark and evaluation measures address gaps in civic QA coverage, failure attribution, and search behavior.

Motivation and evaluation gaps

  • Errors in public-sector guidance can cause significant and irreversible harm, motivating evaluation of LLM accuracy on authoritative information.The introduction gives examples of incorrect chatbot guidance involving employment, food safety, housing, and labor matters.Source: §1 · p. 1
  • CIVI addresses gaps in civic QA benchmarks that lack cross-national and inter-jurisdictional scope, failure decomposition, and systematic study of search invocation and authoritative citations.The introduction argues that civic reliability cannot be inferred from one country and that realistic evaluation should use agentic search rather than closed-book testing.Source: §1 · p. 1

CIVI benchmark

  • CIVI comprises 4,097 expert-verified QA pairs from 576 authoritative pages, spanning four COFOG functions, three jurisdictional levels, and Canada, the United States, and Australia.The benchmark jointly covers cross-national and inter-jurisdictional civic QA using an international classification standard; its construction framework is country-agnostic.Source: §1 · p. 2

ARISE failure diagnostic

  • 72.1% of all observed failures across ten models were attributed by ARISE to retrieval-bound causes rather than gaps in parametric knowledge.ARISE combines source-injection ablation with search-trace analysis to assign failures to four mutually exclusive modes: search bypass, retrieval failure, grounding failure, and comprehension failure.Source: §1 · p. 2

Deployment-realistic evaluation

  • CIVI evaluates ten frontier LLMs using overall accuracy, search invocation rate, Selective No-Search Accuracy, and tier-inclusive Authoritative Hit Rate.Authoritative Hit Rate measures how often cited URLs come from authoritative government sources.Source: §1 · p. 2

2 Dataset Construction

CIVI is built from expert-curated public-sector pages across government functions, countries, and jurisdictional levels, then converted into source-grounded civic questions. Human validation retained questions that met the dataset’s criteria.

2.1 Data Source

  • The 4×3×3 design spans four COFOG functions, Canada, the United States, and Australia, and federal, state or provincial, and local levels, yielding 576 curated pages.Three public-sector experts selected 16 substantive, decision-relevant pages per cell from government or government-funded domains.Source: §2.1 · pp. 2–3

2.2 Question Generation

  • Questions generated from the pages must be single-page derivable, consequential, jurisdiction-specific, and supported by verbatim source quotations.They test required actions or substantive service features, excluding trivia, definitions, and common-knowledge questions.Source: §2.2 · p. 3
  • Each QA pair has eight options, a correct option set, a gold source URL and page title, supporting quotations, and COFOG and jurisdiction tags.Distractors come from controlled page-content perturbations or plausible but unsupported civic misconceptions, deduplicated by underlying misconception.Source: §2.2 · p. 3

2.3 Human Validation

  • 94.1% of 4,354 generated pairs were accepted by both validators, who matched on 96.8% of pairs.Two domain-expert researchers independently reviewed every pair; either validator’s failure flag led to exclusion.Source: §2.3 · p. 3

3 Evaluation Protocol

The protocol evaluates agentic access to civic information using inference-time search, ten frontier LLMs, and metrics for correctness and search and sourcing behavior.

3.1 Evaluation Setup

  • The agentic condition uses inference-time search through Exa, allowing up to three search rounds and at most 10 retrieved URLs.Each round allows unlimited queries; Exa provides a common search substrate for comparing search behavior.Source: §3.1 · pp. 3–4

3.2 Models

  • Ten frontier LLMs represent likely public-sector deployments: seven closed-source models across three provider families and three open-weight models.The closed-source set includes default consumer-facing models and higher- and lower-tier variants; the open-weight set spans major open-source families.Source: §3.2 · p. 4

3.3 Metrics

  • Four metrics assess correctness, search invocation, selective no-search behavior, and authoritative sourcing.Accuracy requires an exact, order-invariant, case-insensitive match to the correct option set; search invocation rate measures the share of questions with at least one search, overall and by jurisdictional level.Source: §3.3 · p. 4
  • Selective No-Search Accuracy is a per-model diagnostic, while ∆skip measures its advantage over disabled-search accuracy on the full benchmark.For authoritative hit rate, eligible government domains depend on jurisdiction: federal questions admit federal domains, state questions state and federal domains, and municipal questions municipal, parent-state, and federal domains; meanAHR averages per-question proportions of cited URLs from authoritative domains among questions with at least one citation.Source: §3.3 · p. 4

4 CIVI

Across CIVI, none of the ten frontier LLMs matched the attentive human baseline. Agentic search improved average accuracy over closed-book answering and single-retrieval RAG, while models differed in search behavior and authoritative-source citation.

4.1 Results

  • 78.28% (95% CI ±1.3 pp) was Gemini 3.1 Pro’s top full-benchmark accuracy, 14.4 percentage points below the five-user mean of 92.67%; no model matched the attentive human baseline.On the subset answered by human participants, Gemini 3.1 Pro scored 77.33%, 15.3 percentage points below the human baseline.Source: §4.1 · p. 5, Table 2 · p. 5
  • Mean accuracy declined from 65.6% on federal questions to 62.6% on state/provincial and 56.7% on local/municipal questions, an 8.9-point federal-to-local difference.The authors hypothesize this pattern aligns with sparse representation of authoritative local sources in general-purpose training corpora.Source: §4.1 · p. 5, Figure 1 · p. 5
  • Federal-to-municipal accuracy changes ranged from +1.9 percentage points for GPT-5.4 to −22.5 for Claude Haiku 4.5, while accuracy ranges widened from 26.4 points federally to 38.1 municipally.The GPT family dropped less than 5 percentage points across levels; the authors hypothesize that increased search invocation at lower levels contributes to robustness.Source: §4.1 · p. 5
  • Gemini 3.1 Pro’s lead was 5.4 percentage points at state level (80.3% versus GPT-5.4 at 74.9%) and 10.5 points federally (82.2% versus Claude Sonnet 4.6 at 71.7%).Across countries, equal-weighted mean accuracy was 63.4% for U.S., 62.0% for Australian, and 59.1% for Canadian questions; the 4.3-point country range was smaller than the 8.9-point federal-to-local difference.Source: §4.1 · pp. 5–6

4.2 Agentic Search Behavior

  • Search invocation ranged from 13.5% for Gemini 3 Flash to 99.8% for DeepSeek V3.2, and federal-to-municipal invocation gaps ranged from 23.4 points to 62.0 points.Search behavior varied independently of open-source versus proprietary status and generally increased at lower jurisdictional levels.Source: §4.2 · p. 6, Table 3 · p. 7
  • Agentic search improved average accuracy by 10.0 percentage points over closed-book answering and 9.1 points over single-retrieval RAG, which improved on closed-book by less than 1 point.Among seven models with sufficient no-search answers, six had positive ∆skip values; Mistral Large 3 was highest at +12.9 points, while Gemini 3 Flash was −3.7 points.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 Authoritative Source Citing

  • Models split into meanAHR tiers of 0.69–0.71 and 0.63–0.65, with overlapping confidence intervals within each tier.A meanAHR of 0.63 indicates that roughly 63% of a model’s cited URLs are authoritative on the average question.Source: §4.3 · pp. 6–7, Figure 2 · p. 6
  • MeanAHR rank showed insufficient evidence of correlation with general capability rank (Spearman ρ = 0.27, p = 0.45, N = 10; Kendall τ = 0.24, p = 0.38).Three of the top five meanAHR models were GPT variants, while Gemini 3.1 Pro led aggregate accuracy but ranked in the lower meanAHR tier.Source: §4.3 · p. 7

5 Decomposing Failures

CIVI introduces ARISE to assign every incorrect response exactly one of four mutually exclusive failure modes. Across ten agents, search bypass and retrieval failure together accounted for 72.1% of failures on average.

Diagnostic methodology

  • ARISE classifies every incorrect response as search bypass, retrieval, grounding, or comprehension through source injection and analysis of the original search trace.With search disabled and the gold source page supplied, a still-incorrect answer is classified as comprehension; otherwise, the original trace distinguishes no search invocation (search bypass), failure to retrieve the gold URL (retrieval failure), or retrieval of that URL despite an incorrect answer (grounding failure). The trace was logged during the original evaluation, so no additional model calls were needed for trace analysis.Source: §5 · pp. 7–8, Figure 3 · p. 8

Retrieval-bound failures dominate

  • 72.1% of failures on average across ten models were retrieval-bound: search bypass was 34.5% and retrieval failure 37.6%, ranging from 48.9% (DeepSeek V3.2) to 92.9% (Gemini 3 Flash).The failures occurred when agents either did not invoke search or did not surface the authoritative source.Source: Figure 4 · p. 8, §5 · p. 8

Comprehension errors persist

  • 2.3%–4.8% of each model’s failed questions remained incorrect even with the gold source page supplied in context.These comprehension errors cannot be fixed by better retrieval and may reflect the difficulty of public-sector institutional language, including precise terminology and complex conditions.Source: §5 · p. 8, Figure 4 · p. 8

Diagnostic implications

  • ARISE’s four failure modes map to distinct intervention targets: search triggering, authoritative-source retrieval, answer-evidence consistency, and reasoning support or human escalation.These are implications of the diagnostic categories, not interventions evaluated in the study.Source: §5 · pp. 8–9

6 Related Work

CIVI extends civic-information benchmarking across countries, jurisdictional levels, and functional categories, while ARISE applies source-injection ablation and search-trace analysis to diagnose agent failures.

Failure Decomposition

  • Prior failure analyses use trajectory taxonomies, claim metrics, and pipeline decompositions; ARISE uniquely combines source-injection ablation with search traces to identify bypass, retrieval, grounding, and comprehension failures.Mallen et al. (2023) distinguish parametric from nonparametric memory, while Wu et al. (2025) classify agentic search as over-search or under-search.Source: §6 · p. 9

LLM Evaluation for Civic and Public-Sector Information

  • Existing civic-information benchmarks focus on single-country, single-jurisdictional scopes (Majithia et al., 2026; Harris et al., 2025; Gao et al., 2026; Liu et al., 2025; Yang et al., 2025; Wang et al., 2025).Source: §6 · p. 9
  • CIVI is the first benchmark to jointly span cross-national and inter-jurisdictional civic QA under an international functional standard, covering three democracies, three levels, and four COFOG categories.Prior work also evaluates municipal open-data and regulatory knowledge (Siebenmann et al., 2026; Guha et al., 2023).Source: §6 · p. 9

7 Conclusion

CIVI diagnoses search-agent failures in civic information across government contexts and uses ARISE to locate failures by stage. None of the ten evaluated agents matches the attentive human baseline, while 72.1% of failures are retrieval-bound.

7 Conclusion

  • CIVI is the first framework for diagnosing civic-information search-agent failures, spanning three federal democracies, three jurisdictional levels, and UN-standard functional categories.The benchmark combines cross-national and interjurisdictional government contexts.Source: §7 · p. 9
  • ARISE combines source-injection ablation with agentic search-trace analysis to distinguish search bypass, retrieval, grounding, and comprehension failures.This localization indicates whether interventions should target search invocation, authoritative-source retrieval, answer-evidence consistency, comprehension support, or human escalation.Source: §7 · p. 9
  • ARISE attributes 72.1% of failures to retrieval-bound causes, comprising 34.5% search bypass and 37.6% retrieval failure; none of the ten evaluated agents matches the attentive human baseline.Source: §7 · p. 9
  • Civic deployers should consider information-change speed, jurisdictional level, and whether answers must be traceable to authoritative government sources, alongside accuracy.Source: §7 · p. 9

Limitations

CIVI evaluates English-language civic information in Canada, the United States, and Australia, and routes all models’ agentic search through Exa. Broader language and governance coverage and testing with models’ native search infrastructure remain future directions.

Geographic and linguistic scope

  • The evaluation covers English-language civic information in Canada, the United States, and Australia; expansion requires expertise in each country’s languages and administrative system.Canada is officially bilingual, but the source pages and questions are English only; extending CIVI to multilingual QA and different legal or administrative traditions is a future direction.Source: Limitations · p. 9

Uniform search substrate

  • All models use Exa for agentic search, prioritizing controlled comparison; future work could test native search infrastructure to assess deployment-specific ecological validity.Source: Limitations · p. 9

Ethical Considerations · A Dataset Format

CIVI documents its human-participant safeguards and dataset release terms, then describes a curated source index and a single-source QA-instance format. Its examples show how source-page evidence supports civic questions.

Ethical Considerations

  • The five volunteer human-baseline participants were recruited from the authors’ personal networks and received no compensation.Participation was voluntary, participants could withdraw at any time, and all gave informed consent after receiving study and task information; no personally identifiable or sensitive information was collected, and the task posed minimal risk.Source: Ethical Considerations · p. 9

A Dataset Format

  • CIVI is released under CC BY 4.0, and its accompanying code under the MIT License.Source: A Dataset Format · p. 13
  • The underlying source pages come from government materials available through open-government or comparable open-access provisions.The stated examples include the Open Government Licence in Canada and Australia, U.S. federal public-domain works, and comparable state and municipal provisions.Source: A Dataset Format · p. 13

A.1 Source Page Curation

  • The source index organizes pages by country, jurisdictional layer, and institution, assigning each page to one COFOG category; it covers 576 pages across 36 strata.Source: §A.1 · p. 13, Figure 5 · p. 14

A.2 Example QA Instance

  • Each evaluation instance is grounded in one curated source page; the example asks how to obtain an Australian Nationally Coordinated Criminal History Check.The example identifies the Australian Government as the federal institution and cites its “Apply for a police check” page.Source: §A.2 · p. 13
  • The example’s source evidence says the ACIC does not accept individual applications; a check can be obtained through an accredited body or an Australian police agency.Source: §A.2 · p. 13

A Dataset Format

  • 4,097 QA pairs span 36 cells formed by 4 COFOG categories, 3 countries, and 3 jurisdictional levels.Source: Table 4 · p. 14

B Comprehension Error Example · C Additional Robustness Analyses · C.1 Matched Comparison on Human Questions

The example shows GPT-5.4 selecting an incorrect option despite reproducing the source’s eligibility criteria. On the 150-question matched subset, Gemini 3.1 Pro remains below the human baseline, while matched-subset accuracy is close to full-benchmark accuracy across models.

B Comprehension Error Example

  • The source requires supervision and eligibility based on age 13 or under, or age 14–18 with a disability.These conditions map to options B and C.Source: B Comprehension Error Example · pp. 14–15
  • GPT-5.4 reproduced the correct conditions in its explanation but its final selections did not reflect them.The analysis identifies this mismatch as a comprehension failure under source injection.Source: B Comprehension Error Example · pp. 14–15
  • GPT-5.4 selected C and H, missing B even though the source’s “13 or under” criterion maps directly to B.H incorrectly describes the condition as ages 13–18 with supervision.Source: B Comprehension Error Example · pp. 14–15

C Additional Robustness Analyses · C.1 Matched Comparison on Human Questions

  • The matched comparison evaluated models against the human baseline on the same 150 questions.This is the matched subset examined in the robustness analysis.Source: §C.1 · p. 15

C.1 Matched Comparison on Human Questions

  • 15.3 percentage points below the human baseline, Gemini 3.1 Pro was the strongest model on the matched subset.Table 5 compares each model with the human baseline on those questions.Source: §C.1 · p. 15
  • 1.8 percentage points was the mean absolute difference across models between matched-subset accuracy and full-benchmark accuracy.The comparison is between each model’s accuracy on the same 150 questions and on the full benchmark.Source: §C.1 · p. 15

C.2 Country Results … D.1 QA Pair Generation

The appendices report country-level accuracy averaged across models, measure run-to-run stability, and document how QA pairs were generated and validated.

C.2 Country Results

C.3 Stability Across Three Runs

  • Only Gemini 3 Flash and Qwen3.6-Plus exchanged order across runs; their full-benchmark confidence intervals in Table 2 overlap.Source: §C.3 · p. 15
  • 2.2 percentage points was the mean within-model run range, and the largest was 3.4 percentage points across three runs on the same 410-question subset.Table 7 reports accuracy across the three runs.Source: §C.3 · p. 15, Table 7 · p. 16

D.1 QA Pair Generation

  • Claude Opus 4.6 generated all QA pairs at temperature 0, using each full source page in one call without chunking or retrieval.Source: §D.1 · p. 16
  • The model received the institution, country, jurisdictional level, and page text, then returned JSON with 3–10 eight-option questions per page and verbatim supporting quotations.Source: §D.1 · p. 16
  • Structural and quotation checks discarded malformed questions and pairs with missing or unmatched source quotations; failed page outputs were retried at most three times.Source: §D.1 · p. 16
  • 4,354 QA pairs remained for human validation after the structural and quotation checks.Source: §D.1 · p. 16

D.2 Evaluation · D.3 Search Baseline Conditions · E Source Page Curation Criteria

CIVI evaluates ten models under bounded Exa agentic search and compares them with search-disabled and single-retrieval baselines using exact-match scoring. Its source pages were curated from official government sources by public-sector experts and used to generate multiple-choice question–answer pairs.

D.2 Evaluation

  • Each agent could issue up to three Exa search rounds and retrieve at most ten URLs while answering at temperature 0.All ten models were accessed through OpenRouter.Source: §D.2 · p. 16
  • Answers were scored correct only when Gemini 2.5 Flash’s extracted option set exactly matched the gold set, and manual checks confirmed all 100 sampled extractions.Queries, retrieved and cited URLs, and final responses were logged for each question.Source: §D.2 · p. 16

D.3 Search Baseline Conditions

  • The search-disabled and single-retrieval RAG baselines used the same ten model versions, 4,097 questions, response format, temperature, and exact-match scoring as the agentic condition.Search-disabled models received no retrieved context; RAG issued the verbatim question once to Exa and inserted the top three results, without reformulation or further retrieval. Table 3 reports 95% Wilson score confidence intervals.Source: §D.3 · p. 16

E Source Page Curation Criteria

  • The 576 source pages came exclusively from official government domains, and three public-sector experts curated them against four selection criteria.Domains included national, state, provincial, and municipal government sites and accredited subdomains.Source: E Source Page Curation Criteria · pp. 16–17
  • For each of 36 jurisdiction-function cells, an expert mapped the responsible government body during curation and gathered candidates from its official website, with at least four times the 16-page target.All three experts reviewed and finalized the pages for every cell together.Source: E Source Page Curation Criteria · p. 17
  • Pages had to be authoritative, substantively informative, relevant to residents’ decisions, and free of known restrictions on AI training or evaluation.Retained pages had a median length of roughly 1,000 words; shorter pages were kept only when dense with resident-relevant information. A small number were excluded for AI-use restrictions.Source: E Source Page Curation Criteria · p. 17
  • The fixed target was 16 pages per cell, selected from the most substantive and representative qualifying pages when candidates exceeded that number.One expert led candidate gathering, and all three experts reviewed and finalized every cell’s selection.Source: E Source Page Curation Criteria · p. 17
  • A prompt asked question writers to generate multiple-choice QA pairs from each curated page’s full text, with the number based on page content.The prompt set minimum and maximum question counts using placeholders filled at generation time.Source: E Source Page Curation Criteria · p. 18, Figure 6 · p. 18

Related papers

Loading 2609.08094v1…