Source-linked AI summary
Designing a Boundary Negotiating Artifact for Collaborative Socio-Technical Sense-Making in AI Regulatory Sandboxes
Idoia Landa-Oregi, Tom Deckenbrunnen, Alessio Buscemi, Daniele Pagani, German Castignani
TL;DR
AI Regulatory Sandboxes provide a legal process for AI assessment, but stakeholders with different expertise still lack shared understanding. Using Research-through-Design, the paper iteratively develops a tool and positions it as a framework backbone for collaborative sense-making. The design process identifies differences in how stakeholder groups understood the tool, while sustained use in a live sandbox remains unevaluated.
Problem
AIRS research emphasizes legal and institutional design, while practical stakeholder sense-making remains a gap despite growing AI assessment demand and limits in current practices.
Source: §1 · p. 1, §2.3 · p. 5Method
Research-through-Design iteratively developed a stakeholder collaboration tool and used its design findings to structure an AIRS framework.
Source: §4.1 · p. 7, §6 · p. 12Results
The tool’s touchpoints form a framework backbone for stakeholder sense-making, while first-iteration feedback found it better understood by technical experts than less technical users.
Source: §6 · p. 12Takeaways & Limitations
AIRS operationalization requires artifacts that help stakeholders make sense of information together, not simply more documentation or guidance.
Source: §6 · p. 12The framework is a design proposal based on one evolving tool and stakeholder feedback, not sustained longitudinal use in a live AIRS.
Source: §6 · p. 13
Abstract
from arXiv · showhide
The rapid, unpredictable advancements in AI system capabilities has seen regulators take adaptive and experimental approaches to policymaking. Established in other domains as instruments balancing regulation with innovation, regulatory sandboxes are seen as solutions for AI regulation. However, analyses mostly focus on the legal and institutional design of AI Regulatory Sandboxes (AIRSes). With the legal framework leaving the socio-technical interpretation to stakeholders, this creates a gap on the sense-making required to fulfill the AIRS purpose. In this paper, we approach this by designing a Boundary Negotiating Artifact as a way to mediate meaning in AIRSes. Through Research-through-Design we iteratively develop a tool, providing an interface for the different stakeholders to collaborate in AI assessment. We then position it as technical backbone in established AIRS frameworks, structuring the collaborative sense-making of the involved stakeholders. We further report the insights gained from our design process leaving the qualitative evaluation for future work.
1 Introduction
AI Regulatory Sandboxes provide a legal process for collaboration, but stakeholders still need to make sense of AI systems and regulation across different expertise and vocabularies. This paper uses Research-through-Design to develop a tool that mediates stakeholder collaboration and structures AIRS sense-making.
1 Introduction
- AIRS legislation creates space for stakeholder collaboration, but leaves innovators and regulators without the shared vocabulary and sense-making infrastructure needed to navigate a sandbox encounter.Innovators translate technical systems into compliance language, while regulators assess unfamiliar systems against abstract legal provisions.Source: §1 · pp. 1–2
- AIRS research has focused on legal and institutional design, leaving practical communication and shared understanding among participating stakeholders underexplored.Prior work describes Boundary Negotiating Artifacts as tools for communicating stakeholder perspectives and reducing information asymmetry, but their design depends on understanding users’ experiences and needs.Source: §1 · p. 2
- The proposed tool structures stakeholder collaboration in AIRSes and is intended to reduce information asymmetry.The paper positions it as a technical Boundary Negotiating Artifact within established AIRS frameworks.Source: §1 · p. 2
- The paper applies Research-through-Design to understand regulators’ and innovators’ experiences and develop a tool that mediates their interaction.The investigation also includes technical experts and asks what design knowledge can inform AIRS operationalization.Source: §1 · p. 2
2 Research context
AIRS frameworks establish legal and institutional procedures, while practical coordination and shared sense-making across stakeholders remain challenging. The section situates the proposed tool in this context, including the varied needs of innovators, technical experts, and authorities.
2.1 Legal and Institutional Design of AIRS in Europe
- AIRS legal and institutional scholarship describes sound procedures, but leaves practical communication and shared understanding among participants underexplored.Boundary negotiating artifacts support collaborative negotiation of meaning and stabilization of boundary objects in heterogeneous collaborations.Source: §2.1 · p. 3, §2.2 · p. 4
- EU law requires national AIRSes but permits variation in sector, expertise, and infrastructure, while procedural harmonization does not address stakeholders’ day-to-day sandbox experience.The resulting fragmentation motivates a shared tool and process structure.Source: §2.1 · p. 3
2.2 Boundary Negotiation in Multidisciplinary Collaborative Work
- Boundary objects and negotiating artifacts support collaboration across groups with different expertise, while AIRSes require technical frameworks such as collaborative dashboards and role-specific access.In AIRSes, AI systems and applicable regulations serve as boundary objects for negotiation.Source: §2.2 · pp. 3–4
2.3 Human Experience of Stakeholders in AIRS
- AI assessment demand exceeds current practices, and use-case-specific risks, metrics, tests, and thresholds make a common technical sandbox unsuitable.Innovators struggle with regulatory complexity and documentation; technical experts manually assemble tests and results, while authorities may lack AI assessment expertise.Source: §2.3 · pp. 4–5
- Assessment results require cross-stakeholder interpretation because technical metrics need use-case context and may not capture legal implications.The same 99 % accuracy can be appropriate for a security camera but unacceptable for pedestrian detection in an autonomous car.Source: §2.3 · p. 5
3 Methods
The study uses Research-through-Design to iteratively develop a tool for stakeholder interaction in AI Regulatory Sandboxes. Human-Centered Design phases and stakeholder feedback inform the tool and the framework built from the design process.
3 Methods
- Research-through-Design guides the study through iterative artifact development, with research questions supporting a framework for stakeholder interaction in AIRSes.The work develops the framework from findings gathered while designing a tool to facilitate configuration of AI Technical Sandboxes.Source: §3 · p. 5
- The tool’s development follows Human-Centered Design phases of understanding, defining, developing, and evaluating solutions through iterative feedback.Stakeholders are engaged in each iteration to address user needs while maintaining viability.Source: §3 · p. 6
- Stakeholder expectations were adjusted across iterations, and the tool served as a basis for a framework describing stakeholder roles and exchanges through its touchpoints.The framework focuses on the core phases of testing and evaluation.Source: §3 · p. 6
4 Designing a computer-assisted BNA for AIRS
The paper develops a digital tool as a Boundary Negotiating Artifact (BNA) to support collaboration in AIRS assessment, refining it through three design iterations. Stakeholder feedback shaped a more inclusive tool and the framework interactions it supports.
4 Designing a computer-assisted BNA for AIRS
- The tool was designed to facilitate stakeholder exchanges through AI assessment configuration, using a baseline for a framework of AIRS interactions.It provides a catalog of tests, controls, and datasets, and guides selection, test execution, result visualization, and reporting.Source: §4 · p. 6
4.1 Design Conception
- Only technical users readily understood the initial concept, prompting a less technical path for regulators to access results and track testing.This feedback informed the first digital-tool design and stakeholder interactions in the framework's testing and evaluation phases.Source: §4.1 · p. 7
4.2 Building a Demonstrator
- The demonstrator evaluation found a technical-assessment focus, while some users needed help qualifying systems; the next iteration added qualification before test selection and controls for lower-risk systems.The prototype was evaluated through walkthroughs and discussions with stakeholders in two settings.Source: §4.2 · pp. 7–8
4.3 Moving Beyond the Technical
- The final iteration added AI-system qualification and control checklists, with system details and tags supporting the selection of relevant tests and controls.The prototype was presented to stakeholders and released as an open-source tool for technical users to test.Source: §4.3 · p. 8
- Stakeholders appreciated the simplified navigation of testing options and controls, shared results, and harmonized tests that let users learn a testing process once.These findings informed the final framework and the interactions among stakeholders across its phases.Source: §4.3 · p. 9
5 A Technical BNA as Backbone in AIRS Frameworks
The framework positions the tool as a touchpoint for stakeholder interactions throughout an AIRS. It retains a consolidated lifecycle structure and describes how the tool supports collaboration in key phases.
5 A Technical BNA as Backbone in AIRS Frameworks
- The framework positions the tool as the main touchpoint for the potential experiences and interactions of three AIRS stakeholder groups.It aims to provide a common platform for technical and non-technical users and reduce complexity in setting up testing environments and AIRSes.Source: §5 · pp. 9, 11, 12
- The framework synthesizes existing AIRS lifecycle phases rather than proposing new ones, locating the BNA's contribution in how it supports stakeholder interaction.The cited sources describe a sequence from participant selection through preparation, testing, exit, and monitoring.Source: §5 · pp. 9–10
- The six phases are pre-application, application and selection, on-boarding, testing, evaluation, and post-participation; the tool supports four core phases.The framework maps stakeholder actions across the process and shows the tool's role in each phase.Source: §5 · p. 10, Figure 4 · p. 10
- Before application, innovators learn about AIRS benefits and prepare applications; NCAs disseminate information and support preparation, while the tool has no role yet.During on-boarding, innovators and NCAs align on a sandbox plan, and technical experts use the tool to select relevant tests, datasets, and controls.Source: §5 · pp. 10–11
- During testing, stakeholders can view and comment on results in a collaborative dashboard; evaluation includes an automatically generated report for sharing and review.Testing iterations identify system strengths, gaps, and risks; the report can highlight failed test cases for further technical analysis.Source: §5 · p. 11
- Post-participation keeps communication going after formal AIRS participation ends, with stakeholders sharing knowledge and gathering feedback on the process.The framework notes that many multi-stakeholder processes conclude without a dedicated phase for what happens afterward.Source: §5 · p. 11
6 Discussion
AIRSes need shared sense-making, not simply more information: stakeholders hold legitimate expertise but lack a shared understanding. The paper’s technical BNA structures collaboration through lifecycle touchpoints, while its findings remain a design proposal awaiting evaluation in a live sandbox.
6 Discussion
- Stakeholders’ central AIRS challenge is making sense of information across differing expertise, not an information deficit.The paper argues that additional documentation or guidance alone is insufficient; stakeholders need artifacts for negotiating meaning.Source: §6 · p. 12
- The tool’s qualification, tests, dashboard, and report provide distinct sense-making functions across AIRS lifecycle phases.Together, these touchpoints form the framework’s technical backbone and cumulatively reduce information asymmetry.Source: §6 · p. 12
- Designing common ground requires respecting differences in authority, accountability, and literacy while stakeholders interpret the same AI results.The paper extends HCI’s multi-stakeholder design practices into the regulatory context of AIRSes.Source: §6 · p. 12
- First-iteration feedback found the tool easier for technical experts to understand than for less technical users.Stakeholder engagement in each RtD iteration surfaced interaction insights that became visible through participants’ behavior with the prototype.Source: §6 · p. 12
- The framework assigns stakeholder actions at each step to mediate supervision, planning, and testing as meaningful exchanges.It positions the tool as a shared artifact and practical means of operationalizing AIRSes.Source: §6 · p. 13
- Across EU member states, the framework offers NCAs a replicable structure and gives innovators a more consistent reference across sandboxes.The paper says this could avoid designing collaboration tools from scratch and reduce innovators’ cost of learning new collaboration models.Source: §6 · p. 13
- Transfer beyond the EU is presented as a possibility, but differences in institutional capacity and regulatory culture may require modifications.The framework’s touchpoints are organized around stakeholder sense-making needs rather than a particular legal text.Source: §6 · p. 13
- The findings come from one evolving tool and semi-structured stakeholder feedback, not sustained use in a live AIRS.The authors frame the framework as a design proposal validated through iterative exposure; confidentiality also limits publication of raw quotes and case-specific findings.Source: §6 · p. 13
7 Conclusion
The paper concludes that AIRS collaboration requires shared sense-making across stakeholders’ differing vocabularies and expertise. It presents an iteratively developed BNA and framework, while calling for studies in live sandboxes and pilots across EU member states.
7 Conclusion
- AIRS stakeholders’ core challenge is shared sense-making across differing vocabularies and expertise, rather than a lack of information.The paper reports a Research-through-Design process grounded in workshops and three iterations of a digital tool.Source: §7 · p. 13
- The resulting BNA is intended to bridge information asymmetry across the AIRS lifecycle and support stakeholder collaboration.The authors present the framework as a path toward interoperable and human-centered practice as AIRSes multiply across EU member states.Source: §7 · p. 13
- Future work should study the tool under live-sandbox conditions and pilot it across EU member states to examine interoperability and replicability.The proposed pilots would examine how the framework handles procedural variation between NCAs.Source: §7 · p. 14