PaperlayerView exact revision on arXiv

Source-linked AI summary

CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information

Dingying Liu, Yunshun Zhong, Wentao Zhang, Yiyuan Li

arXiv:2609.08094v1cs.AI

TL;DR

CIVI addresses the lack of broad, deployment-realistic evaluation and diagnosis for civic-information search agents, where incorrect guidance can cause serious harm. It benchmarks ten frontier models across governmental contexts, measures search and sourcing behavior, and uses ARISE to localize failures. No evaluated agent matches the attentive human baseline, and ARISE attributes 72.1% of failures to retrieval-bound causes.

  • Problem

    Civic-information evaluation has been limited in cross-national and interjurisdictional scope, while search invocation and authoritative sourcing have not been systematically studied despite the risks of incorrect public-sector guidance.

  • Method

    CIVI evaluates ten frontier search agents across civic contexts and combines accuracy with search invocation, Selective No-Search Accuracy, and Authoritative Hit Rate; ARISE uses source-injection ablation and search traces to classify failures.

  • Results

    None of the ten evaluated agents matches the attentive human baseline, and ARISE attributes 72.1% of observed failures to retrieval-bound causes.

  • Takeaways & Limitations

    CIVI indicates that civic deployers should assess not only accuracy but also search invocation, information change, jurisdictional level, and traceability to authoritative government sources.

  • Takeaways & Limitations

    The evaluation covers English-language civic information from Canada, the United States, and Australia, and all models use a single search provider, Exa.

Abstract

from arXiv · show

Large Language Models are increasingly deployed in public-sector settings, where incorrect guidance can cause irreversible harm. We introduce CIVI, the first framework for diagnosing search agent failures in civic information. Its benchmark instantiation jointly spans cross-national, interjurisdictional government contexts (federal, state, and local) and functional categories from an internationally adopted United Nations standard. We evaluate ten frontier search agents and find that none matches an attentive human baseline. Alongside accuracy, CIVI measures search invocation rate, selective no-search accuracy, and how often agents cite authoritative government sources. To perform this diagnosis, we introduce ARISE, which decomposes agentic search failures into four mutually exclusive modes, isolated via source-injection ablation. ARISE attributes 72.1% of all observed failures to retrieval-bound causes rather than to gaps in the models' parametric knowledge.

1 Introduction

CIVI addresses the need to evaluate civic-information search agents across governmental contexts, jurisdictions, and functions where errors can cause serious harm. It combines deployment-realistic evaluation with ARISE, a diagnostic that attributes search-agent failures to distinct stages.

  • Motivation: Public-sector civic information has near-zero tolerance for error because a single incorrect statement can cause significant and irreversible harm.The paper motivates evaluation using examples of misinformation in benefits, tax, legal, housing, labor, and health contexts.
  • CIVI Benchmark: CIVI benchmarks civic question answering across countries, government levels, and four COFOG-based government-function categories.Its benchmark contains 4,097 expert-verified question-answer pairs grounded in 576 authoritative public-sector pages.
  • CIVI Benchmark: The benchmark jointly spans Canada, the United States, and Australia; federal, state or provincial, and municipal jurisdictions; and an international government-function standard.This design is intended to measure civic reliability beyond any single country or jurisdictional level.
  • ARISE failure diagnostic: ARISE combines source-injection ablation with search-trace analysis to assign failures to search bypass, retrieval, grounding, or comprehension.The four modes are mutually exclusive.
  • Deployment-realistic evaluation: CIVI evaluates ten frontier models with accuracy, search invocation rate, Selective No-Search Accuracy, and Authoritative Hit Rate.The latter measures how often cited URLs come from authoritative government sources.

2 Dataset Construction

CIVI is constructed from expert-curated authoritative government pages and validated, single-page-grounded multiple-choice questions. Its balanced design spans government function, country, and jurisdictional level while enforcing concrete consequences and jurisdictional precision.

  • Data Source: CIVI stratifies its source corpus across four COFOG functions, three countries, and three jurisdictional levels, yielding 36 cells.The functions are Public Order and Safety, Economic Affairs, Health, and Social Protection.
  • Data Source: Three domain experts curated 576 authoritative public-sector pages by selecting substantive, decision-relevant content across the stratification cells.Each cell contributed its strongest retained pages after excluding landing pages and navigation hubs.
  • Question construction: Questions must be answerable from one source page, test a concrete human consequence, concern one jurisdiction, and have exactly one correct answer.These criteria exclude trivia and require questions to address actions or substantive public-service features.
  • Question construction: Each question records its correct option set, authoritative source URL, page title, and verbatim supporting quotations for provenance and validation.Distractors are created through controlled perturbations or plausible unsupported misconceptions.
  • Validation: Two public-sector AI researchers independently validated all 4,354 generated pairs, with 96.8% raw agreement; pairs failing any criterion were excluded.The final dataset therefore reflects independent review against both the source page and the construction criteria.

3 Evaluation Protocol

The evaluation models realistic civic-information access by allowing frontier LLMs to search at inference time through a standardized substrate. It measures correctness together with search behavior and source authority.

  • Agentic setting: All ten evaluated frontier LLMs access civic information through agentic search rather than receiving curated context in advance.The setup reflects how people use publicly available AI assistants to retrieve authoritative sources during answering.
  • Agentic setting: All models use Exa as a common search substrate, with up to three query rounds and a maximum of 10 retrieved URLs.The common provider enables direct comparison of search behavior across model families.
  • Metrics: Accuracy uses exact set match, counting a response as correct only when it selects exactly the correct option set.This avoids treating partially correct selections as sufficient when they could still mislead residents.
  • Metrics: Search invocation rate measures the share of questions on which a model searches, while Selective No-Search Accuracy measures correctness among questions where it does not search.Selective No-Search Accuracy is a model-specific behavioral diagnostic rather than a direct cross-model closed-book measure.
  • Metrics: Authoritative Hit Rate measures the proportion of cited URLs that come from government domains admitted for the question’s jurisdictional level.Federal questions admit federal domains, while state and municipal questions admit progressively broader parent-government domains.

4 CIVI

Across CIVI’s 4,097-question evaluation, frontier models trail attentive humans, and accuracy declines as questions move from federal to local jurisdictions. Agentic search improves accuracy over closed-book and single-retrieval baselines, but search behavior, source authority, and failure patterns vary substantially across models.

  • Accuracy: All ten models trail the attentive human baseline; Gemini 3.1 Pro reaches 78.28%, 14.4 percentage points below it.The matched subset gap is 15.3 percentage points.
  • Jurisdictional variation: Accuracy declines monotonically from 65.6% at federal questions to 56.7% at local/municipal questions.The federal-to-local difference is 8.9 percentage points.
  • Jurisdictional variation: Inter-model accuracy spread widens from 26.4 percentage points federally to 38.1 percentage points municipally.The reported ranges are 55.8% to 82.2% federally and 38.1 percentage points municipally.
  • Jurisdictional variation: The federal-to-municipal accuracy change ranges from +1.9 percentage points for GPT-5.4 to −22.5 for Claude Haiku 4.5.All three GPT-family models drop less than 5 percentage points across levels.
  • Agentic search behavior: Search invocation ranges from 13.5% to 99.8% across models and generally increases toward municipal questions.Federal-to-municipal invocation gaps range from 23.4 to 62.0 percentage points.
  • Agentic search behavior: Agentic search improves average accuracy by 10.0 percentage points over closed-book answering and by 9.1 points over single-retrieval RAG.Agentic search exceeds single-retrieval RAG for every model.

5 Decomposing Failures

ARISE combines source-injection ablation with search-trace analysis to classify incorrect civic answers into four failure modes. Across ten models, retrieval-bound causes dominate, while some errors persist even when the authoritative source is supplied directly.

  • Diagnostic methodology: ARISE classifies failures as search bypass, retrieval failure, grounding failure, or comprehension failure using source injection and trace analysis.Source injection reruns failed questions with search disabled and the gold source page supplied; trace analysis then examines search behavior and retrieval.
  • Diagnostic methodology: Source injection eliminates retrieval, grounding, and search-bypass explanations by design; persistent errors are therefore attributed to comprehension failure.The procedure supplies the gold source page while disabling search and checks whether the answer remains incorrect.
  • Diagnostic results: 72.1% of failures across ten models are retrieval-bound, comprising 34.5% search bypass and 37.6% retrieval failure.The combined retrieval-bound share ranges from 48.9% for DeepSeek V3.2 to 92.9% for Gemini 3 Flash.
  • Diagnostic results: 2.3%–4.8% of each model’s failed questions persist with the gold source page in context, showing that better retrieval cannot eliminate comprehension errors.The supplied source page has a median length of approximately 1,000 words.
  • Diagnostic implications: The four diagnostic modes identify different intervention targets, including search triggering, query reformulation, authoritative retrieval, answer-evidence consistency, and comprehension support.The passage frames these as implications of localization rather than interventions evaluated in this study.

6 Related Work

Prior civic-information benchmarks commonly use single-country or single-jurisdiction scopes. CIVI extends this work by jointly covering countries, governmental levels, and internationally standardized government-function categories.

  • Failure decomposition: Failure-decomposition research has used trajectory-step taxonomies, claim-level metrics, and pipeline-stage decompositions for retrieval-augmented and agentic systems.CIVI’s source-injection ablation is motivated by the parametric-versus-nonparametric memory distinction.
  • Prior civic-information benchmarks: Existing civic-information benchmarks generally evaluate government knowledge within single-country or single-jurisdiction scopes.Related benchmarks also address municipal open-data and regulatory knowledge.
  • CIVI’s scope: CIVI is the first benchmark to jointly span cross-national and interjurisdictional civic question answering under an international functional classification standard.Its instantiation covers three federal democracies, three jurisdictional levels, and four COFOG categories.

7 Conclusion

CIVI provides a cross-context framework for diagnosing search-agent failures in civic information, while ARISE separates failures across retrieval, grounding, and comprehension stages. None of the ten evaluated agents matches the attentive human baseline, and retrieval-bound causes account for most failures.

  • Conclusion: CIVI diagnoses search-agent failures across three federal democracies, three jurisdictional levels, and United Nations COFOG functional categories.ARISE distinguishes search bypass, retrieval, grounding, and comprehension failures.
  • Conclusion: None of the ten evaluated agents matches the attentive human baseline.The comparison is reported as a conclusion of the benchmark evaluation.
  • Conclusion: 72.1% of failures are retrieval-bound, comprising 34.5% search bypass and 37.6% retrieval failure.CIVI links these localized failure stages to intervention targets such as search invocation, authoritative retrieval, answer-evidence consistency, and comprehension support.

Limitations

The evaluation is limited to English-language civic information from Canada, the United States, and Australia, and all models use a single search provider. These choices support controlled comparison but constrain multilingual, cross-system, and deployment-specific generalization.

  • Geographic and linguistic scope: The evaluation covers English-language civic information in Canada, the United States, and Australia, while broader multilingual and governance-system coverage remains future work.Canada’s official bilingual status is not represented because the source pages and questions are English-only.
  • Uniform search substrate: All models access agentic search through Exa, prioritizing controlled comparison but limiting assessment of native search infrastructure.Future evaluations could test models under their native search systems to assess ecological validity in specific deployments.

Ethical Considerations

The benchmark uses curated public-sector sources organized across countries, jurisdictional levels, institutions, and COFOG categories. Its example QA instances preserve source-page provenance and supporting evidence.

  • QA instances: Each evaluation instance is grounded in a single curated source page and identifies its institution, jurisdiction, category, and source URL.The example uses an Australian Government police-check page and records the source page alongside the question.
  • QA instances: The example question provides verbatim source spans supporting the correct application routes, including an ACIC-accredited body or an Australian police agency.The source explicitly states that the ACIC does not accept applications directly.
  • Benchmark design: CIVI comprises 4,097 QA pairs distributed across 36 stratification cells.The cell structure is the product of four COFOG categories, three countries, and three jurisdictional levels.
  • Benchmark design: 576 source pages span 36 strata defined by four COFOG categories, three countries, and three jurisdictional levels.The index organizes pages by country, jurisdictional layer, and institution, assigning each page to one COFOG category.

B Comprehension Error Example

A source-injection example shows GPT-5.4 recognizing relevant conditions in its reasoning but selecting an option that incorrectly merges a prerequisite with an age range. The error demonstrates a comprehension failure despite access to the source text.

  • Source conditions: The source requires supervision for all secondary-school children, plus either age 13 or under or age 14–18 with a disability.The relevant source excerpt states both the supervision prerequisite and the two age-based conditions.
  • Error example: GPT-5.4 selected C and H, although the source-supported conditions were B and C.The model identified the disability condition but replaced the separate “13 or under” condition with an overbroad 13–18 option.
  • Error analysis: The model reproduced the source conditions in its explanation but its final answer omitted option B and treated supervision as part of the age range.The analysis characterizes this discrepancy as a comprehension failure under source injection.
  • Human comparison: On a matched 150-question subset, Gemini 3.1 Pro was 15.3 percentage points below the human baseline.Across models, accuracy on the matched subset differed from full-benchmark accuracy by 1.8 percentage points on average.

C.2 Country Results

The country-results section reports country-level accuracy averaged equally across the ten evaluated models and checks result stability across repeated runs and matched human questions.

  • Country comparison: Country results are averaged equally across the ten evaluated models.The country comparison is reported in Table 6.
  • Run stability: Three runs on the same 410-question stratified subset produced a mean within-model range of 2.2 percentage points and a maximum range of 3.4 points.Only Gemini 3 Flash and Qwen3.6-Plus exchanged order, while their full-benchmark confidence intervals overlapped.

D.2 Evaluation

The evaluation gives ten models bounded agentic search access, scores exact answer-set agreement, and compares this condition with search-disabled and single-retrieval baselines. Questions are generated and curated from authoritative government pages using structured, source-grounded procedures.

  • Agentic evaluation: Each model may issue up to three search rounds and retrieve at most ten URLs in total.Agentic search uses the Exa search API as a tool available during answering.
  • Scoring: A response is correct only when its extracted option set exactly matches the gold set, with no missing or additional letters.A separate Gemini 2.5 Flash model extracts the selected letters, and a 100-question manual check found all extractions correct.
  • Baselines: Search-disabled and single-retrieval RAG baselines use the same ten model versions, 4,097 questions, response format, temperature, and exact-match scoring procedure.The search-disabled condition provides no retrieved context, whereas single-retrieval RAG inserts the top three Exa results.
  • Source curation: Source pages are curated by domain experts from official government or government-funded public-sector domains across 36 country, jurisdiction, and function cells.The curation procedure first identifies the responsible government body and then reviews pages from its official website.
  • Question generation: Generated questions contain eight options, may have multiple correct answers, and end with “(Select all that apply)”.Questions are designed to cover decision-relevant content such as eligibility, documentation, deadlines, fees, processes, channels, and entitlements.
  • Question generation: Distractors use controlled perturbations or plausible misconceptions, with each distractor required to test a distinct misconception.Examples include changing thresholds, flipping conditions, altering deadlines, substituting entities, or reordering process steps.
  • Question design: The evaluation excludes trivial definition questions and questions whose answers are obvious from common knowledge.Questions instead target practical resident interactions and service transparency.
Loading 2609.08094v1…