PaperlayerView exact revision on arXiv

Source-linked AI summary

RADAR: Readiness for AI Discovery and Agentic Reach

Luke Jordan, Tiago C. Peixoto, Manuel Ramos-Maqueda

arXiv:2609.28480v1cs.CYcs.LG

TL;DR

Governments increasingly meet citizens through AI, but have limited ways to assess whether AI can both explain public services and reach them. RADAR measures these capabilities across 166 countries, finding that chat scores exceed agent operability scores in every country. Its results point to distinct policy responses, while the measures have stated scope and scoring limitations.

  • Problem

    Governments lack a way to assess whether AI systems can both interpret public-service information and reach services, despite AI’s growing role in mediating access.

    Source: §1 · pp. 2–3

  • Method

    RADAR evaluates informational legibility through chat responses and agent operability through navigation tasks, using distinct measures across countries.

    Source: §2 · p. 3, §2.1 · p. 5

  • Results

    Across 166 countries, chat scores exceeded mean agent operability by 2.37 points on a 0-to-10 scale, and the gap was positive in every country.

    Source: §3 · p. 13, §6 · p. 30

  • Takeaways & Limitations

    Legibility and operability are distinct, so governments can pursue content improvements for AI answers and platform changes that make services reachable to agents.

    Source: §6 · pp. 30–31

    Chat scores measure verifiability rather than ground-truth accuracy, all prompts were in English, and agent operability tests one service, limiting generalization to broader portal estates.

    Source: §5.7 · pp. 29–30

Abstract

from arXiv · show

Governments increasingly meet citizens through an AI system rather than a website. RADAR (Readiness for AI Discovery and Agentic Reach) measures whether that system works, across 166 countries and on two tasks: whether a chatbot can give a correct, officially sourced, country-specific answer about a public service (informational legibility), and whether an automated agent can reach the service to act on it (agent operability). The central finding is that AI can describe public services far better than it can reach them. In every one of the 166 countries, informational legibility scores exceed average agent operability scores under RADAR's respective measures, and the gap does not shrink with national wealth. Income and language explain only part of the pattern, and several governments perform far better or worse than their resources predict. The two failures have different correlates and different fixes. Whether AI can describe a service is associated with how well a country's main administrative language is represented in web-scale corpora, which a government cannot change quickly. Whether an agent can reach it is associated with the country's national web presence, which a government can change now. Traditional digital-government rankings miss the second problem entirely. RADAR lets governments at any income level see it and offers a concrete agenda to fix it, so that public services are not only described by AI but actually reachable through it.

1 Background: Public services and the new intermediary layer

AI is becoming an intermediary for citizens seeking government information and services, while governments have limited visibility into whether AI can read or reach their services. RADAR addresses this measurement gap by assessing both capabilities.

1 Background: Public services and the new intermediary layer

  • AI systems increasingly mediate citizens’ information seeking, while agents are beginning to mediate actions by navigating websites and interacting with service interfaces.These roles create different demands: legibility depends on useful information, while operability can depend on features such as stable URLs, deep links, rendering, and bot controls.Source: §1 · p. 2
  • RADAR pairs country-scale measurement of AI answers about government services with tests of whether agents can reach those services, addressing a gap missed by conventional e-government indices.Earlier benchmarks covered one jurisdiction or the chat channel alone, or tested general agent navigation without focusing on government portals at country scale.Source: §1 · p. 3
  • Governments have invested in human-facing portals, but their services may still be difficult for AI systems to interpret or for automated agents to access.The paper compares the transition to AI access with the earlier shift to mobile use, when portals could be online but unusable on mobile devices.Source: §1 · p. 3

2 Methodology

RADAR separates AI’s ability to give usable, officially grounded service guidance from an agent’s ability to navigate to a service. It evaluates chat responses and agent navigation through distinct tasks and scoring approaches.

2 Methodology

  • RADAR treats usable, officially grounded guidance and an agent’s ability to navigate to a service as distinct capabilities that require different evaluations.A service can be well documented but difficult for agents to reach, or easy to traverse while its guidance is sparse or outdated.Source: §2 · p. 3

2.1 Informational legibility: chat mode

  • Chat mode evaluates model responses to 20 government-service scenarios across four dimensions: verifiability, specificity, depth, and transparency.Scores are averaged across services and models; prompts were run in English, with resident location supplied in the system prompt.Source: §2.1 · pp. 4–5
  • Chat responses are scored by cross-model LLM judges, with country rankings used to account for systematic differences in judge leniency.The judges differ in average scores, but their country rankings are highly similar; the scoring design balances judge weight in the four-model average.Source: §2.1 · pp. 5–6
  • The chat rubric scores verifiability rather than ground-truth accuracy, so an accurate answer without official sources can receive a lower score.The authors chose this approach because manually establishing ground truth across 20 services and 166 countries was beyond the study’s scope.Source: §2.1 · p. 6

2.2 Agent operability

  • Agent operability measures whether an agent can reach service-specific content, and is a descriptive measure rather than a judgment that every access barrier should be removed.The agent is asked to perform a task as a direct user; scores reflect observed navigation rather than the desirability of a barrier.Source: §2.2 · pp. 6–8

2.2.1 DOM mode

  • DOM mode navigates through website structure and text, using four models and browser automation; runs used a single cloud region and a fixed egress IP.The external vantage point can affect results for portals that serve different content by geographic origin.Source: §2.2 · p. 7, §2.2.1 · p. 9

2.2 Agent operability

  • The agent task is obtaining a birth certificate up to the authentication boundary, with the competent service portal potentially operating at a national or subnational level.The country score therefore reflects one service and may refer to a municipal or provincial portal rather than a national one.Source: §2.2 · pp. 7–8

2.2.2 Visual mode

  • Visual mode navigates using screenshots and browser interaction, capturing failures that DOM-based interaction may miss.The final visual runs pool three models, whereas Chat and DOM pool four, an asymmetry relevant to cross-mode comparisons.Source: §2.2.2 · pp. 9–10

3 Results

Across 166 countries, chat scores exceed mean agent-operability scores, but the size of this guidance-to-reach gap does not systematically vary with national wealth. RADAR also finds substantial differences among modes and countries, including governments that outperform or underperform their EGDI-predicted scores.

3 Results

  • Overall RADAR scores range from 3.04 to 8.18, forming a broad gradient rather than a binary split between digitally capable and incapable states.Estonia ranks first at 8.18; North Korea is lowest at 3.04.Source: §3 · pp. 10, 12, Figure 1 · p. 11
  • Chat has the lowest cross-country standard deviation (0.79), compared with 1.49 for Agent DOM and 1.06 for Agent Visual.Chat models also agree more tightly than agent models, while DOM has the largest cross-country variance and Visual model correlations are lowest.Source: §3 · pp. 12–13, Figure 2 · p. 12
  • 82 countries score higher on Visual than DOM, 83 higher on DOM, and one is effectively equal; the mean Visual-minus-DOM gap is 0.06 points.The results do not support a general claim that visual navigation rescues programmatically inaccessible portals.Source: §3 · p. 13, Figure 3 · p. 14
  • 2.37 points is the mean Chat-minus-Agent Operability Index gap; it is positive in every country, though Estonia reverses under the best-agent-channel definition.The gap is descriptive because the two constructs use different task and rubric structures.Source: §3 · p. 13, Figure 4 · p. 16
  • The gap does not widen with wealth: its rank correlation with log GDP per capita is −0.09 and not statistically significant.The DOM-only correlation is −0.13, also not significant.Source: §3 · p. 13
  • Chat models correlate most tightly, up to 0.84, while Visual models agree least, with pairwise correlations as low as 0.24.DeepSeek’s chat advantage over GPT-5 and Claude falls from 1.13 to 0.42 points after accounting for the judge offset.Source: §3 · p. 17, Figure 5 · p. 18

3.1 Peer-relative performance: who beats their income and digital development

  • Côte d’Ivoire, Togo, Benin, Tanzania, and Ghana outperform their EGDI-predicted overall scores, while Saudi Arabia, the UAE, Iran, Turkmenistan, and South Africa underperform.Ghana’s residual is +0.86; the pattern shows that digital infrastructure does not automatically translate into AI accessibility.Source: §3.1 · p. 19, Figure 7 · p. 21

4 What predicts readiness: language, web presence, and the two channels

The analysis asks whether chat legibility and agent operability are associated with different country characteristics, consistent with relying on different information channels. Administrative-language coverage is associated with Chat, while national web presence is associated with agent operability, with important limits on interpreting these observational results.

What predicts readiness: language, web presence, and the two channels

  • Net of EGDI, administrative-language coverage is associated with Chat legibility (partial r = 0.33), but not DOM or Visual operability (partial r = 0.03 and 0.00).An English-language-portal indicator independently predicts Chat (coefficient +0.34, p < 0.001), but not DOM or Visual.Source: §4 · pp. 22–23
  • Net of EGDI, national web presence is associated with the Agent Operability Index (partial r = 0.30), but not significantly with Chat, in a 59-country sample.The sample is selected on the predictor; the DOM association is significant, while the Visual association is not conventionally significant without the population control.Source: §4 · p. 22
  • Verifiability is the chat-rubric dimension that meaningfully separates countries, falling from 2.04 in high-income countries to 1.04 in low-income countries.Specificity, depth, and transparency differ by 0.40 points or less; the reported failure is often a wrong or dead official link rather than vague answers.Source: §4 · p. 23
  • The associations are modest, a government-domain coverage measure adds essentially nothing beyond EGDI, and English-language chat may confound language coverage with query–content language matching.The national-web-presence association applies to the 59-country subsample where it could be measured; the authors say the bounds do not eliminate the observed divergence.Source: §4 · p. 23
  • EGDI explains about 69 percent of Chat variance; for DOM, government effectiveness explains 26 percent versus EGDI's 24 percent, a suggestive rather than decisive reordering.The Visual mode does not reproduce the reordering, and EGDI remains the largest single coefficient in the combined DOM model.Source: §4 · p. 23
  • Language coverage is associated with Chat, while national web presence is associated with agent operability; the pattern is consistent with different information channels but is observational.The figure reports relationships net of EGDI; the analysis does not experimentally isolate parametric or retrieved knowledge.Source: §4 · pp. 22–23, Figure 8 · p. 24

5 Policy Implications

RADAR frames AI-mediated access as a present service channel and proposes making authoritative government information more legible and services more operable for AI. Its policy agenda combines recurring readiness checks, content and architecture changes, agent-friendly interfaces, and diagnosis tailored to each government’s starting position.

5 Policy Implications

  • Governments should make AI accessibility tradeoffs visible: architectural complexity can block automated systems even when portals work for people.The paper calls this deliberate work of making authoritative content discoverable, structured, and interpretable by AI systems sovereign legibility.Source: §5 · p. 25

5.1 Recognize AI-mediated access as a service channel

  • Automated, repeatable AI readiness checks can monitor access after major changes; the 166-country assessment took weeks at a fraction of traditional measurement costs.Government effectiveness predicts agent operability roughly as strongly as digital-government investment.Source: §5.1 · p. 25

5.2 Improve informational legibility through structured, authoritative content

  • Prioritize authoritative procedural content, then distribute it where models retrieve information; better structured content can also improve SEO, accessibility, and human usability.The paper recommends clear service guidance on eligibility, requirements, fees, timelines, and official entry points, while noting that benefits of structured-content strategies may appear only in future training cycles.Source: §5.2 · pp. 26–27

5.3 Improve agent operability by reducing invisible navigation friction without compromising security

  • Mean DOM-versus-Visual difference is 0.06 points, so use the diagnostic country by country; positive gaps may reflect intended transactional controls.The paper distinguishes fixable navigation friction from security or authorization controls that should remain.Source: §5.3 · p. 27

5.4 Offer agent-friendly interfaces for high-value information flows

  • Documented APIs and standardized tool interfaces can expose authoritative information directly, while personalized determinations and consequential actions require authentication, consent, and authorization.Official sources and verifiable provenance can improve answer substance and traceability without requiring the state to operate every citizen-facing interface.Source: §5.4 · p. 28

5.5 Differentiate strategy by starting position

  • Strategy depends on starting position: sophisticated portals may need an agent-friendly access layer, while lower-capacity governments building anew may benefit from simpler architecture.Estonia, France, and the United States illustrate early standards and API-first design; Ghana outperforms its EGDI-predicted score, though that is not proof of deliberate strategy.Source: §5.5 · p. 29

5.6 Use RADAR diagnostically, not only as a ranking exercise

  • Use RADAR’s components and score gaps to distinguish content problems from navigation or access barriers, then determine whether friction is avoidable or a legitimate control.Responses may include publishing changes, architectural redesign, calibrated access, or retaining a control.Source: §5.6 · p. 29

5.7 Limitations

  • Chat legibility is limited by an English-only prompt protocol, while operability reflects one birth-certificate service and a single external geographic vantage point.The web-presence measure covers 59 of 166 countries; reported cross-country correlations are observational associations, not mechanisms.Source: §5.7 · pp. 29–30

6 Conclusion

Across 166 countries, AI describes public services better than agents can reach them, and the gap is not systematically smaller in poorer countries. The conclusion argues that the two capabilities have distinct correlates and calls for governments to shape how AI represents authoritative services while recognizing limits in the measurement.

6 Conclusion

  • Across all 166 countries, Chat exceeds average operability by 2.37 points on a 0-to-10 scale, and the gap does not reverse under the mean-based operability index.The largest measured gaps are Saudi Arabia (Chat 8.73, operability 3.61) and South Africa (Chat 8.44, operability 3.33).Source: §6 · p. 30
  • Chat correlates only moderately with agent navigation (r = 0.55 with DOM; r = 0.57 with Visual), indicating distinct but related constraints.Chat legibility is associated with language representation in training corpora, while operability is associated with crawlable national web presence.Source: §6 · p. 30
  • Traditional digital-government measures predict legibility better than operability; the proposed sophistication penalty is a hypothesis because portal architecture was not measured.Ghana’s performance above its digital-development prediction is consistent with, but does not prove, simpler architecture being easier for machines to navigate.Source: §6 · p. 30
  • Sovereign legibility means deliberately making authoritative content discoverable, structured, and interpretable by AI; it is a no-regret strategy, not a guarantee.The same machine-readable content can improve search ranking and human usability, but models may still hallucinate or prefer unofficial sources.Source: §6 · p. 31
  • RADAR’s findings are bounded by changing AI capabilities and portals, and proposed extensions include multilingual prompts, manual verification, and longitudinal tracking.The English-only protocol leaves the language finding partly confounded with query-language matching.Source: §6 · p. 31

9 Appendix A. The fuller research agenda

The appendix proposes extensions to strengthen RADAR’s measurement as AI systems and government portals change. Priorities include disentangling language and training effects, refining saturated rubric dimensions, and testing robustness, geographic variation, and real-world relevance.

Disentangling training from retrieval; Multilingual evaluation

  • Repeating chat evaluations in each country’s official language would test whether the Anglophone advantage reflects training-corpus representation or query-language matching.A controlled comparison with search and tool use disabled versus enabled would isolate the parametric contribution.Source: §9 · p. 33

Ground-truth verification

  • Manual verification of a stratified sample could reveal common answer errors and build an error taxonomy for agent-navigation failures.The proposed examples include jurisdiction confusion, outdated requirements, and fabricated steps.Source: §9 · p. 33

Geographic vantage

  • Because agent runs used one cloud region and a fixed egress IP, multi-region and in-country tests could measure dependence on geographic origin.Such comparisons would help distinguish portal design from geographic gatekeeping.Source: §9 · p. 33

Rubric refinement

  • Revise or report rubric dimensions separately: several operability dimensions are saturated, while chat legibility rests almost entirely on verifiability.Structured Access scores zero for 84 percent of observations, Findability is at ceiling for 93 percent, and chat Transparency is at ceiling for 98.9 percent of responses.Source: §9 · p. 33

Prompt robustness; Explicit machine-readability indicators; End-to-end transactional flows; Longitudinal tracking

  • Further checks include controlled prompt paraphrases, explicit machine-readability indicators, and extending tests to transactional flows in sandboxed environments.Transactional measurement would cover application submission, payment, and status tracking; longitudinal measurement could assess whether interventions improve accessibility.Source: §9 · pp. 33–34

Cross-validation with user outcomes; Consistency as an accuracy signal

  • Linking RADAR scores to citizen outcomes could test whether the constructs correspond to welfare-relevant differences; cross-model agreement could also be validated against hand-checked accuracy.Proposed outcomes include error rates, completion time, and satisfaction with AI-mediated guidance.Source: §9 · p. 34

10 Appendix B. Prompts and scoring rubrics

Appendix B records the prompts, scoring rules, and reliability procedures used to generate RADAR scores. It specifies the chat evaluation, service battery, agent task and rubric, and a second-model audit of agent runs.

10 Appendix B. Prompts and scoring rubrics

  • The appendix reproduces prompts and rubrics verified against released run data, including the agent formula that reproduces published totals for every country and model.Two judges scored each chat conversation except those generated by the judging model itself.Source: §10 · p. 34, §10.1 · p. 34

10.1 B.1 Chat mode: LLM-as-judge prompt

  • Chat answers are scored from 0 to 10 for official-source verifiability, specificity and actionability, depth and completeness, and transparency without invention.Judges score only the answer and cited sources; a correct but unsourced answer is penalized, while a verifiable answer that a service is unavailable can score highly.Source: §10.1 · pp. 34–35

10.2 B.2 Chat mode: the twenty-service battery

  • The chat battery uses three-turn resident-style conversations about 20 government services, with all prompts issued in English.The turns state a need, ask about online availability, and probe a specific detail.Source: §10.2 · p. 35

10.3 B.3 Agent mode: operability prompt and rubric (v3)

  • Agents search in the primary official language for an existing birth-certificate copy, using passport application as a fallback when no online birth-certificate service exists.Recorded outcomes include an online form, a directly queryable civil register, or the fallback; high rubric scores require observed evidence and flagged results may be reviewed.Source: §10.3 · pp. 36–37
  • The agent total normalizes a weighted six-dimension raw score to 0.0–10.0, with Findability weighted four times and the other dimensions added once.The dimensions are Findability, Portal Quality, Agent Permeability, Service Access, Structured Access, and Navigation Efficiency.Source: §10.3 · p. 36

10.4 B.4 Reliability auditor

  • A second model reviewed roughly 15 percent of agent runs per batch and adjusted scores when run logs revealed clear errors.Source: §10.4 · p. 37
Loading 2609.28480v1…