Source-linked AI summary
A Reusable Semantic Web Framework for Evidence-Grounded Fundamental Rights Impact Assessments under the EU AI Act
Faith Olopade, Delaram Golpayegani, David Lewis
TL;DR
Evidence needed for credible public-sector FRIAs is fragmented across incompatible incident repositories, risk vocabularies, and legal texts. The paper presents a Semantic Web framework and evaluates it on two high-risk categories; five scenarios cover 68.7% of the corpus, while employment-domain classification achieves near-chance agreement.
Problem
Evidence for credible FRIAs is fragmented across incident repositories, risk vocabularies, and legal texts, making it difficult for deployers to assemble it for assessments.
Source: §1 · p. 2Method
The paper builds a Semantic Web framework for two high-risk Annex III categories, combining a curated corpus with keyword, LLM, and hybrid annotation methods.
Source: §1 · p. 2, §2.1 · p. 2, §2.3 · p. 3Results
Five FRIA scenarios surface 68.7% of the corpus, while employment-domain classification achieves κ = 0.045.
Source: §3.2 · p. 4, §6 · p. 6Takeaways & Limitations
The near-chance employment result has direct implications for automated fairness assessment, and the released artefacts support adoption and extension by regulators, national authorities, and SMEs.
Source: §6 · p. 6The gold standard was produced by a single annotator, and the corpus covers only two of eight Annex III categories.
Source: §4 · p. 5
Abstract
from arXiv · showhide
The EU AI Act (Art. 27) requires deployers of high-risk AI systems to conduct Fundamental Rights Impact Assessments (FRIAs) before deployment, yet the evidence needed for credible assessments is fragmented across incompatible incident repositories, risk vocabularies, and legal texts. We present a reusable Semantic Web-based framework that consolidates this evidence for two high-risk public sector categories: employment and worker management (Annex III(4)) and access to essential public services (Annex III(5)(a)). A curated 150-record corpus is annotated along four axes using keyword, LLM, and hybrid methods and serialised as a SPARQL-queryable knowledge graph of 1,351 RDF triples. Five FRIA demonstration scenarios surface 103 records (68.7% coverage). Evaluation against a 69-record gold standard reveals that LLM-assisted classification of the employment domain achieves only $\kappa = 0.045$, a cautionary result for automated fairness-related evidence retrieval in this domain. All artefacts are released openly to support adoption by regulators, national authorities, and SMEs.
1 Introduction
Evidence for credible public-sector FRIAs is fragmented across incident repositories, risk vocabularies, and legal texts, requiring substantial manual reconciliation. The paper presents an interoperable Semantic Web framework for AI Act FRIAs focused on two high-risk application categories.
1 Introduction
- The Dutch benefit-fraud case and SyRI demonstrate welfare-profiling harms,, while Mobley litigation alleges discriminatory résumé screening.The paper describes a persistent pattern of rights violations when high-stakes public-sector AI risk assessment is inadequate.Source: §1 · pp. 1–2
- Incident evidence, risk vocabularies, and legal obligations are fragmented and incompatible, leaving deployers to reconcile them manually for a given assessment.Incident repositories use free-text narratives and community tags, while existing risk vocabularies are not integrated into FRIA workflows.Source: §1 · p. 2
- The framework connects evidence for AI Act FRIAs across two high-risk Annex III application categories.The extended abstract presents its framework, empirical findings, and implications for algorithmic fairness research and practice.Source: §1 · p. 2
2 Framework
The framework combines evidence from three source types, classifies records along four axes, and represents them using interoperable Semantic Web technologies. Its regulatory crosswalk links two AI Act high-risk categories to Charter rights, while a query interface lets users filter graph records.
2.1 Corpus Construction
- The 150-record corpus combines incident reports, U.S. government AI use cases, and European human-rights cases to cover employment and essential public services.AIAAIC captures real-world failures, the Federal Inventory documents active government deployments, and ECtHR cases provide legal analysis of rights violations.Source: §2.1 · p. 2
- Records are classified by Annex III domain, implicated Charter rights, risk pattern, and causal factor.Rights are multi-label; risk patterns include bias or discrimination, privacy breach, procedural unfairness, lack of transparency, or other, while causal factors include data quality, model design, deployment context, or oversight failure.Source: §2.1 · p. 2
2.3 Annotation Pipeline
- The annotation pipeline compares a synonym-expanded keyword baseline, a zero-temperature few-shot Claude Sonnet classifier, and a hybrid that assigns domains to keywords and rights to the LLM.The hybrid resolves conflicts using priority logic, retaining each method's classifications for the axes where it performs better.Source: §2.3 · p. 3
2.4 Regulatory Crosswalk
- The regulatory crosswalk maps Annex III(4) and Annex III(5)(a) obligations to specific Charter articles, making the mapping explicit and machine-readable.The mapping is not enumerated in the AI Act itself and provides a structured link between regulatory requirements and rights evidence.Source: §2.4 · p. 3
2.5 Query Interface
- The Flask interface lets users filter the knowledge graph by Annex III domain, Charter rights, and risk pattern.It moves the framework toward interactive compliance tooling anticipated by Art. 27(5).Source: §2.5 · p. 3
3 Evaluation
Evaluation finds sharply different domain-classification agreement for employment and essential services, alongside increased risk-pattern coverage from the hybrid method. Five scenarios retrieve records for FRIA use, including a welfare-eligibility example with provenance linking evidence to incident precedents.
3.1 Gold Standard and Agreement Metrics
- The evaluation uses a stratified 69-record gold standard (46% of the corpus), manually annotated from original sources using written criteria for each axis.The subset supports per-axis agreement estimates across both Annex III domains and all three sources rather than exhaustive manual annotation.Source: §3.1 · p. 3
3.2 Domain Classification
- κ = 0.045 for employment versus κ = 0.525 for essential services; incidental labour-market language drives employment overprediction.Employment records often discuss the domain incidentally rather than as the primary harm, and both keyword and LLM classifiers over-predict it.Source: §3.2 · p. 4
- Near-chance employment agreement means automated employment-risk annotation cannot currently be trusted without human review; GPT-4o-mini also yields κ = 0.196 for domain classification.The authors warn that relying on such tools for FRIA evidence may produce an incomplete and misleading evidence base.Source: §3.2 · p. 4
3.3 Coverage and Risk Patterns
- The hybrid method reduces unknown risk-pattern classifications from 61.3% with the keyword baseline to 14.0%, while agreement remains slight-to-fair.Procedural unfairness and lack of transparency show the greatest disagreement, primarily because incidents involve co-occurring failure modes.Source: §3.3 · p. 4
3.4 Use Case: A Welfare Eligibility Deployer
- In the welfare-eligibility scenario, the graph returns 57 candidate records with risk and causal-factor classifications and provenance to originating incidents.The use case traces flagged non-discrimination risks to precedents including the Dutch childcare-benefits scandal and SyRI.Source: §3.4 · p. 4
3.3 Coverage and Risk Patterns
- 103 of 150 records (68.7%) are surfaced across five FRIA scenarios, with welfare-eligibility and cross-domain profiling retrieving the largest pools and recruitment the fewest.Figure 1 presents per-scenario retrieval; the scenarios may retrieve overlapping records.Source: §3.3 · p. 4, Figure 1 · p. 5
4 Limitations
The current evaluation and corpus impose limits on the framework's scope and unsupervised use, especially for employment classification.
4 Limitations
- The gold standard has a single annotator, so multi-annotator reliability remains necessary before pipeline outputs can be used without human review.The corpus covers only two of eight Annex III categories, and coverage of biometrics, law enforcement, and migration is untested.Source: §4 · p. 5
- AIAAIC's English-language and U.S./U.K.-weighted sources introduce geographic bias in an EU regulatory context.The schema is designed for extension, but the corpus covers only two Annex III categories.Source: §4 · p. 5
- Employment classification quality is insufficient for unsupervised deployment and requires human oversight for FRIAs relying on that domain's evidence.Source: §4 · p. 5
5 Relevance to ECAF
The paper connects its framework to ECAF’s policy, computer-science, and social-science areas through reusable FRIA infrastructure, annotation evaluation, and a cautionary employment-domain result.
Policy and Law
- The regulatory crosswalk and FRIA scenarios provide reusable infrastructure for Article 27 compliance and contribute to emerging standardisation of AI incident reporting.The paper also frames its legal analysis as a foundation for a possible AI Office FRIA tool and complementary to GDPR-versus-AI-Act impact assessment analysis.Source: §5 · p. 5
Computer Science
- The annotation pipeline, multi-method comparison, and gold-standard evaluation provide evidence on LLM-assisted classification reliability in a regulatory context.The authors identify direct implications for auditing framework design.Source: §5 · p. 5
Social Sciences
- κ = 0.045 marks near-chance employment-domain agreement, cautioning practitioners building automated fairness assessment tools in that domain.The corpus documents real-world AI failures in employment and essential services, domains where automated systems bear directly on the rights of marginalised groups.Source: §5 · p. 6
6 Conclusion
The conclusion presents a Semantic Web framework for consolidating AI risk evidence into a SPARQL-queryable knowledge graph for EU AI Act FRIA compliance. Five demonstration scenarios cover 68.7% of the corpus, while evaluation finds near-chance employment-domain classification and the artefacts are openly released.
Conclusion
- Five FRIA demonstration scenarios cover 68.7% of the 150-record corpus, while gold-standard evaluation finds near-chance LLM-assisted employment-domain classification.The authors identify this finding as having direct implications for automated fairness assessment in a domain associated with algorithmic discrimination.Source: §6 · p. 6
- The framework consolidates fragmented AI risk evidence into a structured, SPARQL-queryable knowledge graph supporting EU AI Act FRIA compliance.The framework is intended to support extension to further Annex III categories.Source: §6 · p. 6
- Schema artefacts and annotated data are released under CC BY 4.0, and pipeline source code under the MIT Licence, to support adoption by regulators, national authorities, and SMEs.Source: §6 · p. 6