Source-linked AI summary
Co-constructing sociotechnical AI governance: participatory system mapping using algorithm registers
Íñigo de Troya, Maurus Enbergs, Neelke Doorn, Roel Dobbe
TL;DR
Algorithm registers may not provide meaningful transparency about the sociotechnical systems governing public-service algorithms. This study combines stakeholder engagement, participatory system mapping, and STPA to examine a Dutch municipal register, finding that it raised more questions than it answered about the tool and its governance.
Problem
It remains unclear how algorithm registers represent embedded sociotechnical systems and provide transparency that supports accountability for affected publics.
Method
The study uses interviews, surveys, and participatory mapping with diverse stakeholders to inform a System-Theoretic Process Analysis of a municipal decision-support tool.
Results
The algorithm register raised more questions than it answered because its information about the tool and governance was too abstract for relevant public actors.
Takeaways & Limitations
Effective accountability requires registers aligned with different publics' information needs and participatory system-theoretic methods engaging civil-society organisations and ombudsmen.
Takeaways & Limitations
System maps impose an ontological framing and cannot fully convey the human stories or concerns of affected citizens.
Abstract
from arXiv · showhide
Algorithm registers have been championed as a means of providing transparency on the use of algorithms in public services. Yet potential publics differ in their expectations of what should be made transparent and how, as well as in their interest in and ability to parse the information currently published in the registers. Moreover, it remains unclear how these instruments can represent the sociotechnical systems in which these algorithms are embedded, and how system-level transparency can facilitate accountability. In this paper, we ask, what do algorithm registers reveal (and occlude) about the sociotechnical systems governing algorithmic systems, and how can diverse stakeholder perspectives inform a more pluralistic system-theoretic safety analysis? To do this, we probe the municipal algorithm register of a Dutch city through a case study of a decision-support tool for caseworkers' assessment of citizens' welfare benefits eligibility based on legal automation through a business rule engine. Through interviews, surveys, and participatory system mapping workshops (with municipal staff, civil society organisations, and ombudsmen, N=8), we seek to understand to what extent the register allows stakeholders to map the algorithmic system in question. These maps inform a System-Theoretic Process Analysis (STPA) that situates the register within a wider sociotechnical governance structure. Participants' contributions allow us to identify potential safety hazards which would not have been possible to see using the algorithm register alone, including benefits eligibility denial, system performance deterioration, and inability to contest wrongful decisions. By engaging both direct and indirect stakeholders, we reflect on the normative dimensions of algorithm governance efforts and how politics shape the practice of system safety analysis.
Introduction
The introduction frames algorithm registers as transparency instruments whose public value and system-level adequacy remain contested. It presents a participatory study examining what registers reveal or occlude and how diverse stakeholders can inform sociotechnical safety analysis.
- Problem: Opaque algorithmic systems can obscure harmful or erroneous decisions and frustrate public scrutiny, while high-profile cases demonstrate the need for transparency and accountability.Meaningful transparency remains elusive, especially when informing the general public.
- Background: Municipal algorithm registers have become popular local-government transparency instruments, including for welfare-benefit eligibility, but their ability to meet diverse transparency needs remains uncertain.The introduction identifies registers as a prominent mechanism for informing citizens about governmental algorithm use.
- Motivation: Registers may nevertheless exert a disciplinary effect by prompting organisations to inventory algorithmic systems, identify risks, and establish mitigation strategies.This secondary use can improve internal awareness of algorithms and their governance.
- Research questions and approach: The study asks what algorithm registers reveal or occlude about algorithmic systems and governance, and how diverse stakeholder perspectives can inform pluralistic system-theoretic safety analysis.It uses workshops, interviews, and surveys with civil society organisations, Ombudsmen, and municipal staff.
- Contributions: The paper contributes an empirical register assessment, a validated participatory schema for mapping algorithmic systems and governance, and reflection on their normative dimensions.The schema integrates perspectives, insights, and needs from direct and indirect stakeholders.
Background & Related Work
Algorithm transparency is contested and relational: its accountability benefits depend on who receives, can access, and can understand disclosed information. Algorithm registers and participatory approaches can broaden system-level analysis beyond expert-led safety assessment by involving intermediaries, civil society organisations, Ombudsmen, and affected communities.
- Transparency and accountability: Transparency’s contribution to accountability depends on actors capable of processing disclosed information, whose access and interpretive abilities differ.Transparency is described as socially situated and relational rather than merely a performative act.
- Transparency and accountability: Algorithm registers seek to legitimate public-sector algorithmic systems, but controversial welfare-profiling systems may remain contested despite such legitimization.Transparency is framed in algorithm-governance discourse as an instrumental means toward accountability.
- Intermediated transparency: Intermediaries such as oversight authorities and societal watchdogs may be better positioned than citizens to interpret registers and support accountability.Third parties can bridge affected individuals’ lived experience of algorithmic harm and the technical expertise needed to unpack it.
- System-level analysis: Algorithm registers can represent sociotechnical systems across technical, operational, organisational, and institutional layers, including risks, discretionary operators, recourse channels, and governance instruments.This whole-system perspective supports analysing and orchestrating safety mechanisms across multiple levels using system-theoretic principles.
- Participatory safety analysis: Participatory approaches can complement safety engineers’ expertise, while civil society organisations and Ombudsmen remain underexplored in participatory AI governance and safety science.Safety science is characterised as largely expert-led, involving stakeholders already capable of contributing expert knowledge.
Case study: the Avola welfare eligibility
The case study examines Avola, a no-code business rule engine that helps municipal caseworkers determine welfare-benefit eligibility by implementing written law. The municipal register describes Avola’s technical, operational, organisational, and institutional components, including datasets, caseworker discretion, complaints procedures, information requests, and implemented legal articles.
- Avola welfare eligibility: Avola is a no-code business rule engine that implements written welfare-benefit eligibility laws for municipal caseworkers.Its law-codification implementation is validated by legal experts.
- Avola welfare eligibility: Caseworkers use Avola to support decisions about whether citizens are eligible for welfare benefits, while retaining discretion over final eligibility decisions.The register also provides information about complaints procedures and Freedom of Information Requests under the Dutch Open Government Law.
- Avola welfare eligibility: The register describes Avola’s technical, operational, organisational, and institutional components.It includes the datasets used for eligibility determination and the legal articles implemented through Avola.
Methods
The study used a five-stage design, focusing on participatory system mapping and subsequent STPA informed by earlier familiarisation with the algorithm register and Avola. It engaged municipal staff, ombudsmen, and civil society organisations through structured workshops to identify components, governance relationships, and safety concerns.
- Study design: The five-stage study focused its reported findings on Stages IV–V, building on earlier familiarisation with the algorithm register, Avola, and methods used for participatory system mapping.Stages I–III prepared participants for the later mapping work.
- Participants: Three stakeholder groups—municipal staff, municipal ombudsman, and civil society organisations—were selected to capture plural views on Avola and the algorithm register.System developers were not engaged because the primary focus was the register rather than Avola.
- Workshops: Each stakeholder group participated in a separate 3-hour mapping workshop producing audio recordings, annotated system maps, and worksheets.A 20-minute STPA introduction familiarised participants with the method before mapping began.
- Participatory mapping: Participants assessed an author-drafted map of register-revealed Avola components, adding publicly documented elements and using 11 worksheet questions to identify omissions.They first worked individually for 40 minutes before exchanging their observations.
- STPA analysis: Participants focused on STPA Step 2, mapping the system, and their input was then used to inform a full STPA of the sociotechnical governance structure.STPA proceeds iteratively from losses and hierarchical control structures to unsafe control actions and loss scenarios.
Participatory System Mapping
Participatory workshops combined six individually produced maps into one multifaceted representation of Avola’s governance architecture. Stakeholders contributed distinct concerns, expertise, and questions shaped by their roles and perspectives on the system, governance, and citizens.
- Participatory System Mapping: Six system maps, one per participant, were combined into a multifaceted representation of Avola’s governance architecture and participants’ socially situated understandings.The combined map illustrated the diversity of participants’ concerns and expertise.
- Civil Society Organisations: Civil society participants focused on technical aspects, governance instruments, and complaints channels, expanding attention beyond the municipal channel identified in the register.Their contributions drew on experience with civil advocacy regarding algorithmic systems and digital rights.
- Civil Society Organisations: Civil society participants questioned who conducted or informed the DPIA and FRAIA, how often they were performed, and whether they remained relevant to the deployed software version.They also noted uncertainty about when the assessments occurred and whether they informed one another.
- Municipal Staff: Municipal staff highlighted political bodies’ governing roles and added institutional nuance absent from the register, including aldermen’s responsibility for commissioning Avola and the register.They also identified reporting relationships involving the register team and Avola’s product owner.
- Ombudsman Staff: Ombudsman staff contributed least because mapping did not fit their human-interest perspective, but they added support organisations and counsellors and used the map to question disclosure sufficiency.Their questions included whether citizens could understand which data were used and which department handled certain matters.
System safety analysis
The STPA broadens analysis from Avola’s technical operation to its operational, organisational, and institutional governance contexts. It reveals hazards involving automation bias, flawed data, weak complaint processes, and insufficient documentation for accountability.
- System safety analysis: The STPA uses participatory mapping to implicate actors and components across technical, operational, organisational, and institutional levels.Loss Scenarios connect these increasingly broad sociotechnical frames in the full safety analysis.
- System safety analysis: A human-in-the-loop does not ensure safety because caseworkers may fail to detect errors or feel compelled to accept outputs perceived as objective.The register’s depiction of caseworkers as final decision-makers rests on strong assumptions about their ability to understand and challenge algorithmic outputs.
- System safety analysis: Out-of-date citizen data can trigger technical and operational errors when automation bias and an organisational culture of trust lead caseworkers to miss failures.The scenario links failed BRP/Suwinet checks to casework errors and undue trust in legal automation.
- System safety analysis: The register leaves complaint handling unclear, while untracked error metrics may prevent systematic patterns from being aggregated and analysed.Incident reports shared with Bizzomate are also insufficiently specified to show whether they concern individual cases or technical logs.
- System safety analysis: Insufficient FRAIA, DPIA, testing, and performance documentation can prevent Ombudsmen and civil-society organisations from scrutinising implementation or substantiating legal claims.The register states that assessments occurred without adequately explaining them, and available performance evidence may remain inadequate even after a Freedom of Information Request.
Discussion
The discussion finds that algorithm registers are valuable accountability instruments but fall short as standalone mechanisms for public oversight. Participatory mapping and indirect-stakeholder scrutiny enrich registers and safety analysis, while participation constraints and political forces shape what they can reveal.
- Limits of registers: Algorithm registers fall short of enabling effective public oversight, despite their ambition to make algorithmic systems transparent and accountable.This limitation applies both to stakeholders focused on citizen impacts and to those concerned with algorithmic inner workings and governance.
- Limits of registers: Municipal capacity constraints and limited evaluation documentation can make declared risk assessments provide the public with a false sense of security.The documentation was accessible through a Freedom of Information Request, but the municipality lacked capacity to facilitate broad participation.
- Participatory governance: Stakeholder input significantly enriched the information available in the register, while scrutiny by Ombudsmen, civil society organisations, and researchers exposed its affordances and limitations.The register describes safeguards such as FRAIA and DPIA assessments, but stakeholder contributions supported reconstructing the wider governance architecture.
- Participatory governance: Mapping functioned as an intervention that enabled participants to question one another and experts, revise safety assumptions, and scrutinise algorithmic systems.The authors therefore encourage safety experts to treat mapping as an end in itself, not merely as preparation for STPA.
- Politics and system safety: STPA can articulate why contextual factors matter for risk mitigation, but its system descriptions also impose a particular ontology on sociotechnical systems.What the register accounts for is shaped by socially situated actors and political forces, while external stakeholders must advocate through political channels to expand its scope.
Conclusion
Algorithm registers can support transparency and accountability, but must better address diverse publics’ information needs and be situated within broader sociotechnical governance practices. Their design and documentation are political, reflecting negotiations over expertise, roles, and institutional power.
- Conclusion: Algorithm registers are an important step toward transparency in public-service algorithm use, while their institutional design remains an opportunity for diverse stakeholder engagement.The conclusion notes unresolved challenges, including criticism that registers lack a clear audience and sufficient force.
- Conclusion: Effective accountability requires aligning algorithm registers with different publics’ information needs and integrating them with existing governance practices.STPA can help reveal this integration and guide participatory critical reflection involving civil society organisations and ombudsmen.
- Conclusion: Mapping and documentation practices are inherently political because registers reflect negotiations among actors with different roles, expertise, and institutional power.This raises the question of who has a say in what gets documented and requires understanding algorithmic systems across technical, operational, organisational, and institutional contexts.
Appendix A: Interview protocols · Interview protocol regarding the algorithm register
The interview protocol examines the municipality’s role, the intended purposes of its algorithm register, and its relationship to the national register. It also asks how well current objectives are being met, what the register does well, and what challenges affect its desired future state.
- Interview protocol regarding the algorithm register: The protocol begins by asking interviewees about their role at the municipality.
- Interview protocol regarding the algorithm register: It asks what the municipal algorithm register is intended to achieve.
- Interview protocol regarding the algorithm register: The protocol examines whether the municipal register’s objectives match those of the national register.
- Interview protocol regarding the algorithm register: Interviewees are asked how extensively the municipal register meets objectives stated in the national register.
- Interview protocol regarding the algorithm register: The protocol asks participants to identify what the register currently does well.
- Interview protocol regarding the algorithm register: It concludes by addressing the register’s desired future state and the challenges involved in reaching it.
Interview protocol regarding the Fundamental Rights Impact Assessment (FRAIA)
The interview protocol examined participants’ roles in the municipality, involvement in the Avola system’s FRAIA, insights gained, subsequent actions, and how unanswered questions were handled.
- Interview protocol regarding the Fundamental Rights Impact Assessment (FRAIA): The protocol asked interviewees about their municipal roles and who else participated in the Avola system’s FRAIA.It also asked what insights the FRAIA provided, including missing or developing change-management, monitoring, steering, and accountability instruments.
- Interview protocol regarding the Fundamental Rights Impact Assessment (FRAIA): Interviewees were asked whether actions followed the FRAIA and what they did when unable to answer a question.
Appendix B: Worksheet
The worksheet guides participants to locate themselves within the system map, identify connections, omissions, assumptions, and sources of knowledge, and evaluate the map’s limitations. It also asks how mapping can support understanding of governance structures and potential negative impacts.
- Participants are asked to locate themselves on the map and describe the components to which they are connected and how.
- The worksheet prompts participants to identify missing components or connections and other elements revealed through their presence on the map.
- Participants are asked which parts of the map need more detail, what abstractions or assumptions they made, and what complaints-procedure information is missing.
- The worksheet asks participants to describe experiences informing their mapping and additional resources they would use besides the register.
- Participants are asked to assess limitations of the mapping approach, suggest improvements, and explain its value for understanding governance and potential negative impacts.
Appendix C: Hierarchical Control Structure
Figure 2 presents part of the municipality’s hierarchical control structure surrounding the Avola system, within its broader design and governance structure.
- Hierarchical Control Structure: Figure 2 details the municipality’s hierarchical control structure.The figure depicts this structure as part of the design and governance arrangements around the Avola system.
- Hierarchical Control Structure: The depicted control structure belongs to the municipality.The passage identifies the structure as municipal.
- Hierarchical Control Structure: The structure forms part of the design and governance structure around the Avola system.Figure 2 situates the municipal control structure within these wider arrangements.