PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLEAF: A Socio-Technical Framework Bridging Custodians and Researchers for Trustworthy Data Sharing

Hibiki Ito, Chia-Yu Hsu, Hiroaki Ogata

arXiv:2610.02720v1cs.CY

TL;DR

Educational real-world data offer research opportunities, but ethical, regulatory, and governance constraints limit sharing, and practical arrangements for access have received less attention. ReLEAF operationalises two-stage sharing and is refined through three design-science cycles; the authors conclude it offers a concrete framework and transferable design knowledge, while identifying limits to evaluation and deployment.

  • Problem

    Practical operational design for ERWD sharing, particularly arrangements between data custodians and researchers, has received less attention than anonymisation and related concerns.

    Source: §1 · p. 1

  • Method

    ReLEAF combines differentially private synthetic data for exploration with controlled real-data validation on request, refined through three design-science cycles.

    Source: §1 · pp. 1–2

  • Results

    Through three cycles, ReLEAF was iteratively refined and formatively evaluated, yielding a concrete operational framework and transferable design knowledge for semi-trusted data access.

    Source: §6 · p. 14

  • Takeaways & Limitations

    ReLEAF offers practitioners a framework for accessing otherwise inaccessible ERWD and highlights further socio-technical research needs.

    Source: §6 · p. 14

    Evaluation generalisability is limited by small samples, one reviewer, and a Japanese setting with few professional researchers; wider real-world evaluation remains needed.

    Source: §5.3 · p. 14

Abstract

from arXiv · show

Growing volumes of educational real-world data (ERWD) are being collected across learning platforms and institutional systems. Sharing these data within the Learning Analytics community offers substantial research opportunities, yet access remains limited by ethical, regulatory and governance constraints. Prior work has primarily focused on anonymisation techniques, but little attention has been paid to operational design of practical ERWD sharing, particularly how data custodians and researchers interact through privacy-preserving access mechanisms. To address this gap, we propose ReLEAF, a socio-technical framework that bridges data custodians and researchers by operationalising two-stage data sharing: 1) Differentially private synthetic data are shared for exploratory analysis, and 2) controlled real-data validation is performed on demand. Following a design-science research approach, we refine and formatively evaluate ReLEAF through three cycles involving 4 graduate students, 6 researchers, and 90 undergraduate students, respectively. Three design principles emerged through the cycles: P1) position privacy-preserving access mechanisms within the research workflow, P2) make the conditions for acceptable secondary use explicit and actionable, and P3) promote engagement with governance requirements rather than automate compliance decisions. Together, ReLEAF provides a concrete framework for trustworthy ERWD sharing, while the design principles offer transferable guidance for other data-sharing contexts.

1. Introduction

The paper addresses the limited operational design of trustworthy ERWD sharing, proposing a socio-technical framework for semi-trusted researchers. ReLEAF combines exploratory access to differentially private synthetic data with controlled real-data validation, and is refined through three design-science cycles.

Introduction

  • Prior work has focused more on anonymisation and stakeholder ethics than on practical arrangements between custodians and researchers for trustworthy data sharing.The paper identifies operational design—not anonymisation alone—as the gap.Source: §1 · p. 1
  • ReLEAF addresses this gap with two-stage access: researchers explore differentially private synthetic data, then request controlled validation on real data.The framework is intended to balance utility and privacy for small-sample, high-dimensional ERWD.Source: §1 · pp. 1–2
  • The target users are semi-trusted researchers: their identities are known, but they are not subjected to months-long screening or given open access.The intended model lets researchers begin exploration promptly while protecting privacy by design.Source: §1 · p. 1
  • The study iteratively develops and formatively evaluates ReLEAF through three design-science cycles built on the LEAF infrastructure.The paper’s main question asks how semi-trusted secondary use of ERWD can be operationalised through a socio-technical framework.Source: §1 · p. 2

2. Literature review

The literature review contrasts trusted, open, and semi-trusted data access, then motivates two-stage sharing as a way to address ERWD privacy–utility tensions. It also distinguishes ReLEAF’s approach from dummy-data workflows that do not reflect real data distributions.

2.1. Data sharing with semi-trusted researchers

  • Trusted access can involve long reviews, require plans before researchers inspect data, and rely on researcher trust after access is granted.The review notes reported access delays of 8–15 months at three UK health-data providers and about six months each for permits and data delivery in a cross-European study.Source: §2.1 · p. 2
  • Open datasets remove approval delays and pre-access planning, but stronger privacy protection can reduce utility; Ito et al. (2026c) reported only 36% successful replication of downstream analyses on real data.The review notes that ERWD are rarely published as open data.Source: §2.1 · p. 2
  • Semi-trusted access combines known researcher identities with no meticulous review of research plans before initial access, while MORF’s restricted environment hindered analysis planning and error handling.MORF limits outputs to common prediction metrics; users may struggle to plan analyses and handle errors without seeing data.Source: §2.1 · p. 2

2.2. Two-stage data sharing

  • Differentially private synthetic data offer a privacy guarantee, but false discoveries and the small-sample, high-dimensional nature of ERWD can worsen the utility trade-off.Two-stage sharing pairs DP synthetic data with controlled real-data validation; although the second stage breaks the overall DP guarantee, its privacy risk can be estimated using Gaussian membership inference privacy.Source: §2.2 · p. 3
  • OpenSAFELY supports script development with dummy data before reviewed real-data validation, but its dummy data do not reflect the underlying distribution.The review suggests this mismatch may matter more in education, where researchers may have less prior knowledge of local, context-dependent data distributions.Source: §2.2 · p. 3

3. Literature-based initial design

The literature-based initial ReLEAF design combines structured data and request organisation with two-stage sharing, publicness-first review, and an LLM-to-human review process. Its criteria focus on technical requirements and whether outputs are suitable for publication, while retaining human accountability.

3. Literature-based initial design

  • The initial design has four components: structured organisation of data and requests, two-stage sharing, publicness-first review criteria, and LLM-to-human review.The organisation covers datasites, datasets, assets, projects, and validation requests, using paired pseudonymised and differentially private synthetic data.Source: §3 · p. 3
  • Researchers use the Lab to download synthetic data and prepare projects and validation requests; custodians manually review requests and run code on datasites through Review.Approved real-data outputs are then made available for download on Lab.Source: §3 · p. 3
  • Review criteria combine technical requirements with a publication-ready requirement intended to discourage individual-level, potentially disclosive outputs.Because synthetic-data findings are group-level trends, the approach seeks to avoid detailed statistical disclosure controls for every finding while making criteria easier to understand.Source: §3 · pp. 3–4
  • Three LLMs independently comment on requests against the proposed criteria, helping researchers revise or submit them while preserving human accountability.The process is intended to reduce manual review time by identifying trivial issues.Source: §3 · p. 4

4. Iterative development and formative evaluation

The study uses design-science research to develop, refine, and formatively evaluate a socio-technical data-sharing artefact across three cycles. It records ethical safeguards and limits participant access to domestic institutions under the stated legal and prototype constraints.

4. Iterative development and formative evaluation

  • Design-science research was chosen to develop and refine a practical socio-technical artefact, with ReLEAF iteratively refined and formatively evaluated across three cycles.The study obtained informed consent from data subjects and participants, and an author with statistical disclosure-control expertise reviewed each validation request.Source: §4 · p. 4
  • Because the system was a prototype and cross-border data sharing was legally restricted in Japan, participants were affiliated with domestic universities or research institutions.Participants were also asked to delete downloaded synthetic data after the studies.Source: §4 · p. 4
  • The study received approval from Kyoto University’s institutional review board, No. 2023-001.Study materials, including instructions, questionnaires, AI prompts, and dataset descriptions, were available in supplementary material.Source: §4 · p. 4

4.1. Cycle 1: Initial user study

Cycle 1 examined when synthetic data alone could support secondary ERWD use and when real-data validation was needed, while testing whether sustained system use clarified review criteria. A five-week study with four graduate students informed the framework’s subsequent design implications.

4.1.1. Methods

  • Four graduate students worked over five weeks on a shared learning-habits project using three annually arriving datasets and differentially private synthetic data.The study aimed to emulate a three-year secondary-use project and included a focus group analyzed using SCAT.Source: §4.1.1 · p. 5

4.1.2. Results

  • 38 requests were created; requests increased over time, but approval rates did not, offering no clear evidence of improved adherence to review criteria.Four requests were rejected and eight withdrawn after LLM reviews.Source: §4.1.2 · p. 5, Figure 3 · p. 6
  • Participants viewed synthetic findings as infeasible for publication or educational practice without guaranteed fidelity, and considered real-data validation necessary for downstream integration.They also noted that synthetic-real discrepancies could damage credibility, transparency in synthesis shaped analysis plans, and privacy review could slow research.Source: §4.1.2 · p. 6

4.1.3. Design implications

  • The study surfaced challenges with human-to-LLM review and criteria comprehensibility; review support should help users improve requests rather than serve as a validation filter.Clarification opportunities and explicit disclosure-risk guidance may address black-box judgments and confusion about publicness-first criteria.Source: §4.1.3 · pp. 6–7

4.2. Cycle 2: Researcher field evaluation

Cycle 2 refined ReLEAF’s chatbot review and disclosure-risk criteria, then explored their use with six active researchers. Results informed revisions to the workflow and review guidance.

4.2.1. Refinement and research questions

  • Cycle 2 replaced mandatory LLM review with a voluntary chatbot offering Review and Dev modes, and added disclosure-risk assessment to the criteria.Review mode supports judgments and follow-up clarification; Dev mode can edit code or draft compliant snippets.Source: §4.2.1 · p. 7, Figure 6 · p. 7

4.2.2. Methods

  • The pilot involved six active users from 17 registered researchers, who used two datasets and completed surveys and request exercises during a ten-week period.The evaluation was exploratory because of its small scale.Source: §4.2.2 · pp. 7–8, §4.2 · p. 7

4.2.3. Results

  • 40 requests were received and 29 approved; around half were submitted without chatbot interaction.The ten-week pilot required participants to download synthetic data and submit at least three validation requests.Source: §4.2.3 · p. 8, Figure 7 · p. 8, §4.2.2 · p. 8
  • Researchers used the AI reviewer to discuss request development and sometimes to check technical and privacy requirements, but one participant reported inaccurate, overconfident judgments.In one case, the chatbot rejected small-cell outputs through several revisions, while a human expert eventually approved the request.Source: §4.2.3 · p. 8
  • Some researchers used the AI reviewer to draft and revise analysis code, suggesting use beyond ReLEAF-specific request creation.For example, it helped P5 cross-tabulate habits and classify students by morning-study time.Source: §4.2.3 · p. 9
  • Initial exercise correctness was 47%; participants also found the two-stage workflow unfamiliar, though post-pilot responses were positive about its comprehensibility and acceptability.Researchers raised concerns about applying the “only necessary information” criterion consistently, while agreeing that the criteria were mostly comprehensive.Source: §4.2.3 · p. 9

4.2.4. Design implications

  • Design implications include positioning both sharing stages in the research workflow, clarifying disclosure risks and project relevance, and making AI review less overconfident.The reviewer should focus on trivial issues while helping users create trustworthy validation requests.Source: §4.2.4 · p. 9

4.3. Cycle 3: Operationalising at scale

Cycle 3 tested ReLEAF’s revised hypothesis-generation-to-validation workflow with undergraduate proxy users and examined request outcomes and human-review burden. Results show active AI-assisted request preparation but limited validation engagement, persistent disclosure-risk challenges, and shorter review times under a revised policy.

4.3.2. Methods

  • The larger-scale study involved 90 undergraduate proxy users completing project and request tasks, with manually reviewed submissions and feedback on synthetic-versus-real results.Researchers used interaction logs, qualitative review of rejected requests, and reviewer logs to examine workflow, issues, and review time; code was not always reviewed in detail when outputs were interpretable from labels and research questions.Source: §4.3 · p. 9, §4.3.2 · p. 10

4.3.1. Refinement and research questions

  • The revisions framed the two stages as hypothesis generation and validation, reorganised review criteria, and asked the AI reviewer to assess categories and return approval, rejection, or unsure.The criteria added restrictions on MIN/MAX queries, project relevance, and numeric-output rounding; the prototype did not require advanced disclosure controls such as small-cell suppression.Source: §4.3.1 · pp. 9–10, Table 3 · p. 9

4.3.3. Results

  • Of 190 approved requests, users explicitly checked validation results for only 124, despite frequent iterative AI interactions and a common request-edit, code-edit, chatbot workflow.Figure 8 visualises the interaction patterns; most requests were submitted after a final AI check.Source: §4.3.3 · p. 10, Figure 8 · p. 11
  • Rejections were 3 of 100 requests (3.0%) for Task 1 and 10 of 103 (9.7%) for Task 2, involving technical errors, weak research questions, dataset misconceptions, and potential disclosure risks.The authors attribute some issues to novice users’ limited data-science experience and note that systematic support could address them.Source: §4.3.3 · p. 11
  • A suppressed coverage-group output still risked revealing that one student belonged to the most inactive group, and the AI reviewer approved it before human review rejected it.The authors note that assessing disclosure risks may challenge both users and the zero-shot AI reviewer.Source: §4.3.3 · p. 11, Table 4 · p. 11
  • Most requests were reviewed within 30 minutes in Cycle 2 and 10 minutes in Cycle 3, with Cycle 3’s shorter review time partly attributable to its code-review policy.The review-time distributions are shown in Figure 9.Source: §4.3.3 · p. 11, Figure 9 · p. 12
  • Input assets, AST nodes, lines of code, and cyclomatic complexity showed unclear relationships with review time, while table outputs took longer in Cycle 3 under the code-review policy.The authors suggest semantic analysis complexity may be more closely linked to review cost, while noting the comparison also reflects Cycle 2 researchers’ greater experience.Source: §4.3.3 · p. 12, Figure 10 · p. 12

4.3.4. Design implications

  • The AI chatbot should remain assistive rather than authoritative, and human review burden should be considered in framework design for wider adoption.The authors also call for technical refinement to reduce inaccurate, overconfident AI reviews.Source: §4.3.4 · p. 12

5. Discussion

The discussion presents ReLEAF as a four-component framework refined through three design-science cycles, with principles for workflow-integrated privacy, actionable secondary-use conditions, and human-accountable governance. It also identifies unresolved limits in review scalability, disclosure-risk communication, evaluation scope, and broader deployment.

5.1. Answer to the main RQ

  • ReLEAF combines the original data and analysis organisation, a hypothesis-generation and validation workflow, technical/disclosure/validity review criteria, and AI-assisted request preparation under human oversight.The authors present these components as a socio-technical framework bridging custodians and researchers for trustworthy secondary use of ERWD.Source: §5.1 · p. 12, Figure 11 · p. 13
  • The three principles are to place privacy-preserving access in the research workflow, make acceptable-use conditions explicit and actionable, and use AI to support governance engagement rather than automate compliance decisions.The authors connect the principles to confusion about stage roles, persistent difficulty interpreting disclosure requirements, and risks of overreliance or inaccurate AI judgements.Source: §5.1 · p. 13

5.2. Implications

  • ReLEAF remains dependent on expert review, and semantic complexity may raise review costs; sustainable human-labour allocation, reviewer training, and reliable risk communication remain unresolved.The authors identify these challenges as potential barriers to scalability and adoption.Source: §5.2 · p. 13
  • ReLEAF does not automatically connect custodians to data subjects or eliminate informed-consent requirements, whose applicability must follow local privacy regulations.Evaluating consent needs for ReLEAF-based ERWD sharing is outside this work’s scope.Source: §5.2 · pp. 13–14

5.3. Limitations and future work

  • The evaluations have limited generalisability due to small samples, one reviewer, and a Japanese regulatory context with few professional researchers; wider real-world evaluation is needed.Future work also includes cross-datasite sharing, automating expert-dependent DP-SDG work, supporting non-LEAF platforms, computational privacy-risk assessment, and managing DP privacy budgets for repeated releases.Source: §5.3 · p. 14

6. Conclusion

The paper presents ReLEAF as a socio-technical framework for trustworthy secondary use of ERWD, refined and formatively evaluated through three design-science cycles. It offers practitioners a framework for accessing otherwise inaccessible ERWD while identifying unresolved challenges and the need for further research.

6. Conclusion

  • ReLEAF bridges data custodians and semi-trusted researchers for trustworthy secondary use of educational real-world data.The framework and its design principles emerged from three iterative design-science cycles with formative evidence.Source: §6 · p. 14
  • The work contributes a concrete operational framework and transferable design knowledge for semi-trusted data access.Source: §6 · p. 14
  • Human-review scalability, disclosure-risk communication, AI calibration, and broader institutional deployment remain challenges for ReLEAF.The authors call for further research on the socio-technical operationalisation of ERWD sharing.Source: §6 · p. 14
Loading 2610.02720v1…