Source-linked AI summary
What Does AI Success Look Like? A Mixed-Methods Analysis of Supranational AI Governance Instruments
Delaram Golpayegani, Marta Lasek-Markey, Monique Munarini, Arjumand Younus, Aphra Kerr, Dave Lewis
TL;DR
Supranational AI governance instruments have articulated success criteria, but early nonbinding policies and guidelines had limited practical effect. This paper analyzes a 13-document corpus using thematic analysis and topic modelling; it finds shared expectations alongside differences in governance approaches and criteria, and recommends greater coordination.
Problem
Supranational bodies have issued AI-governance instruments, but many early policies and guidelines are nonbinding, and AI-success visions differ.
Source: §1 · pp. 1–2Method
The study combines expert-led thematic coding of a representative document subset with BERTopic topic modelling and interprets the findings through the Sociology of Expectations.
Source: §4.2 · p. 6Results
Institutions share expectations that trustworthy AI is achievable and requires multi-stakeholder governance, but differ on governance mode and criteria, with national and regional contexts shaping success criteria.
Source: §7 · p. 11Takeaways & Limitations
The authors recommend operationalizing values, improving interoperability and coordination, and strengthening accountability across the AI value chain.
Source: §7 · pp. 11–12
Abstract
from arXiv · showhide
In the last decade and in particular since 2018, supranational institutions have published policies, guidelines, and most recently regulations to govern development and use of Artificial Intelligence (AI). This paper aims to investigate the themes and dimensions of ``AI success'' set forth by instruments of AI governance issued by supranational bodies. To this end, we use a mixed-methods approach, combining manual thematic analysis and automated topic modelling approaches, to analyse a curated corpus of 13 documents issued by supranational bodies, including the European Union bodies, OECD, G20, UNESCO, United Nations, and the Council of Europe between 2018-2025. Based on the analysis and through the lens of sociology of expectations, we propose a novel categorisation scheme for AI success themes emerging from supranational bodies' documents. Drawing on the findings, we advance a set of policy recommendations, setting forward the need to move beyond high-level principles toward more coordinated approaches to AI governance.
1 Introduction
Supranational AI instruments have shaped expectations about AI success, but many early policies and guidelines were nonbinding. The paper analyzes governance documents from 2018–2025 to identify success criteria and inform recommendations on accountability and coordination.
1 Introduction
- Nonbinding supranational instruments helped shape AI-success expectations, but their lack of direct legal obligations limited their practical effect.The EU AI Act is presented as a binding response, using a risk-based approach.Source: §1 · pp. 1–2
- The study combines manual thematic analysis of a representative subset with topic modelling across documents, interpreting codes and topics through the Sociology of Expectations.The authors treat the documents as expressions of competing social imaginaries shaping AI innovation and governance.Source: §1 · p. 2
- The authors contribute a curated, open-access corpus of 13 AI documents issued by prominent supranational bodies between 2018 and 2025.Source: §1 · p. 2
- Integrating inductively generated codes with computationally extracted topics yielded 157 unique concepts.These concepts are the combined output of the two analytical approaches.Source: §1 · p. 2
- The authors organize AI-success criteria into four meta-themes and ten constituent themes using a micro–meso–macro framework.The framework distinguishes scales at which the expectations are addressed.Source: §1 · p. 2
- The paper proposes policy recommendations addressing accountability and coordination gaps across the AI value chain.The recommendations are informed by analysis of supranational documents.Source: §1 · p. 2
2 Multi-Level AI Governance
The section distinguishes governance across levels and regulatory modes, then surveys how EU and non-EU instruments differ in their legal force and approach.
2 Multi-Level AI Governance
- AI governance spans local to global levels and statutory, co-regulatory, and self-regulatory modes; the paper also links accountability to lifecycle responsibility, auditability, and redress.The paper adopts a definition of AI governance that includes initiatives addressing risks and seeking to harness AI’s potential.Source: §2 · pp. 2–3
2.1 The European approach to AI Governance
- The EU AI Act binds actors operating on the EU market and uses risk categories ranging from prohibited and high-risk systems to limited- and minimal-risk systems.The section describes corresponding restrictions, monitoring, oversight, and transparency obligations; systemic-risk general-purpose AI models receive specific regulation and monitoring.Source: §2.1 · p. 3
2.2 International (Non-EU) Approaches to AI Governance
- International instruments vary in legal force: OECD/G20 and UNESCO recommendations are nonbinding, while the signed Council of Europe Convention remains unratified.The OECD and UNESCO recommendations are addressed principally to governments; UNESCO’s document was signed by 193 countries, and the Convention would become binding upon ratification.Source: §2.2 · p. 4
3 Related Work
Prior research has examined AI-policy documents through thematic analysis, topic modelling, and mixed methods, but the authors identify a gap in systematic analysis of supranational instruments from 2018–2025.
3 Related Work
- The authors report no prior systematic study specifically analyzing supranational AI policies and guidelines across 2018–2025.Earlier descriptive reviews identified recurring principles, while later work organized them into thematic groupings and hierarchies.Source: §3 · pp. 4–5
- Related work uses manual thematic analysis, statistical topic modelling, and increasingly mixed-methods combinations to examine AI governance documents.Examples include NVivo coding, BERTopic and LDA analyses, and studies combining automated analytics with manual analysis.Source: §3 · pp. 4–5
- The present study extends the authors’ earlier EU-focused analysis by including international supranational policies for comparison with EU institutions.The authors describe this broader corpus as offering a more comprehensive view of the global AI-governance landscape.Source: §3 · p. 5
4 Methodology
The paper analyzes a 13-document corpus of supranational AI instruments published from 2018 to 2025, combining manual thematic analysis and topic modelling. Experts interpret the integrated findings and classify themes by the governance level at which they are expected to be addressed.
4.1 Corpus Creation
- The corpus includes formal AI policies, guidelines, and regulations published by major supranational bodies from January 2018 through August 2025.It contains 13 documents and includes both binding and non-binding instruments.Source: §4.1 · p. 5
- Soft governance instruments dominate the corpus, with the EU AI Act as its only regulatory instrument; EU bodies issued 61% of the documents.The corpus is published as an open resource following FAIR data principles.Source: §4.1 · p. 5
4.2 Combining Thematic Analysis and Topic Modelling
- The analysis combines manual thematic analysis with unsupervised topic modelling, then has a multidisciplinary team interpret their integrated results to identify AI success themes.The overall methodology combines the two approaches to identify codes and topics from the documents.Source: §4.2 · pp. 5–6, Figure 1 · p. 6
- Manual coding covers six of the 13 documents, while BERTopic identifies prevalent topics in each document for expert semantic labelling.Two law and computer science experts independently labelled topics through consensus-building; source code and results are archived for reuse and reproducibility.Source: §4.2 · p. 6
- Experts iteratively interpret codes and topics into themes, then classify each theme as micro, meso, or macro according to its expected governance scale.Micro refers to individuals or organisations, meso to sectoral or national policy, and macro to supranational governance.Source: §4.2 · p. 6
5 Codes and Topics
Manual coding and BERTopic identify recurring concepts and topic patterns across the supranational documents. The results show a shift in EU discourse toward regulatory enforcement, alongside broader governance emphases in non-EU documents and 26 topics shared across the two groups.
5.1 Codes from Manual Thematic Analysis
- 75 unique concepts emerged from manual coding of six documents, with 18 concepts appearing in all six.Figure 2 highlights codes found in five or six of the manually coded documents.Source: §5.1 · p. 7
5.2 Topics from BERTopic
- EU documents yielded 100 topic labels, shifting from earlier ethical and trustworthy AI principles toward enforcement, risk classification, and technical compliance in later documents.The later emphasis is most marked in the AI Act and its accompanying guidelines.Source: §5.2 · p. 8
- Non-EU documents yielded 58 topic labels, commonly covering data governance, infrastructure, human rights, sustainable development, cooperation, inclusion, and AI research.The authors characterize these emphases as reflecting the broader normative mandates of bodies such as UNESCO, the UN, and the Council of Europe.Source: §5.2 · p. 8
- The EU and non-EU documents share 26 topics, but non-EU texts give distinctive attention to AI risks, innovation, literacy, global cooperation, inclusion, sustainability, climate, and peace.OECD, G20, and UN topics align on many points, while UNESCO and the Council of Europe diverge; EU discourse foregrounds compliance and enforcement, whereas non-EU texts more often use AI governance language.Source: §5.2 · pp. 8–9, Figure 3 · p. 8
6 AI Success Themes
The integrated analysis organizes 157 concepts into four meta-themes and constituent themes, framed by the governance level at which expectations are addressed. The categories distinguish technical qualities, AI applications and benefits, risks and harms, and governance mechanisms and principles.
6 AI Success Themes
- The analysis groups 157 concepts into four meta-themes—AI technical features, AI uses, AI risks and harms, and AI governance—with constituent themes framed at micro, meso, and macro levels.These levels indicate whether expectations are addressed by organisations or individuals, sectoral or national policy, or supranational policy spanning domains.Source: §6 · p. 9
- AI technical features cover design choices such as computing resources and model training, plus quality characteristics such as transparency, explainability, and robustness, expected to be addressed at micro level.The paper associates these technical design choices with potential economic and societal impacts.Source: §6 · pp. 9–10
- AI uses distinguishes sectoral application areas, such as healthcare and education, from broad benefits such as digital transformation and the digital economy.The former is categorized as meso-level; the latter as macro-level because it can require cross-government approaches and interaction with other governments.Source: §6 · p. 10
- AI risks and harms separates organisationally manageable risks such as discrimination and confidentiality from shared harms such as democracy and human dignity that span meso and macro levels.The shared category reflects risks and harms not yet consistently contained within specific sectors or policy domains.Source: §6 · p. 10
- AI governance spans all three levels, from organisational risk management and testing to sectoral oversight and macro-level rights and governance principles.The macro themes distinguish legally protected rights, including intellectual property and workers’ rights, from coordinating principles such as trustworthy AI and AI literacy.Source: §6 · p. 10
7 Discussion and Policy Recommendations
Supranational documents share an expectation that trustworthy AI is achievable and require multi-stakeholder involvement, but differ on governance modes and criteria. The discussion identifies persistent coordination, accountability, interoperability, and sustainability gaps and recommends moving beyond high-level principles.
7 Discussion and Policy Recommendations
- The documents share an expectation that trustworthy AI is achievable and that multi-stakeholder involvement is necessary, but differ on self-regulation versus co-regulation and on governance criteria.They also broadly expect AI benefits to be realised when protections against risks and harms are in place; EU documents frame this as “innovation-with-protections.”Source: §7 · p. 11
- AI success criteria are not universal: national and regional contexts shape the importance and sensitivity attached to impacts on rights and values such as privacy, autonomy, and non-discrimination.Source: §7 · p. 11
- The regulatory turn leaves gaps between high-level values and their operationalisation, and between jurisdictions; recommendations include technical standards, benchmarks, metrology, regulatory sandboxes, and interoperability.Source: §7 · p. 11
- Accountability is less prevalent in later EU documents than in the 2018 high-level expert group principles, while gaps remain for data providers and model developers and structural power asymmetries.The authors recommend stronger supranational oversight, transparency obligations for big technology providers, and coordination among states.Source: §7 · p. 11
- Shared terms such as “trustworthy” and “humancentric” can express divergent normative commitments, creating ambiguity that undermines alignment and mutual recognition.The authors call for explicit values and shared vocabularies and taxonomies, including use of SKOS.Source: §7 · p. 11
- Persistent fragmentation and limited coordination can hinder responses to systemic risks to fundamental rights; the authors recommend interoperability, sustained multi-stakeholder collaboration, and accountability across the AI value chain.Source: §7 · p. 12
- Environmental impacts remain underrepresented across the corpus, especially in recent EU AI policy documents relative to other supranational documents.Recommendations include sustainability impact assessments, efficiency standards, and alignment with climate objectives.Source: §7 · p. 12
- The recommendations collectively call for more coordinated AI governance that addresses existing gaps and emerging challenges.Source: §7 · p. 12
8 Conclusion
The paper analyzes 13 supranational AI policies, guidelines, and regulations from 2018–2025 using mixed methods. It derives four meta-themes and 10 themes, and identifies expansion of the corpus as future work.
8 Conclusion
- The inductive analysis yielded four meta-themes—AI Technical Issues, AI Uses, AI Risks and Harms, and AI Governance—and 10 distinct themes.Source: §8 · p. 12
- The paper examines how key supranational bodies articulate AI success through mixed-method analysis of 13 policies, guidelines, and regulations published between 2018 and 2025.Source: §8 · p. 12
- Future work aims to expand the corpus to national-level policies, broader regulatory frameworks with indirect AI implications, and technical AI standards.Source: §8 · p. 12