PaperlayerView exact revision on arXiv

Source-linked AI summary

PEDAL: Open Infrastructure for Citable AI Prompts in STEM Education and Research

Murat Kahveci

arXiv:2610.00001v1cs.DL

TL;DR

Educational prompts often remain ephemeral despite their role in instructional practice, and the field lacks infrastructure to treat them as scholarly artifacts. PEDAL provides a citable prompt pipeline and NExAIE applies it to AI-augmented peer review; initial deployment recorded 4,684 artifact views from 1,810 researchers within about one month.

  • Problem

    Prompts embody pedagogical reasoning but remain largely ephemeral and lack standard mechanisms for versioning, citation, or reproduction as scholarly instruments.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    PEDAL makes pedagogical prompts citable scholarly artifacts through a Laboratory-to-Archive pipeline, and NExAIE applies the infrastructure to AI-augmented peer review.

    Source: §1 · p. 2

  • Results

    4,684 lifetime artifact views from 1,810 unique researchers were recorded approximately one month after the first pedagogical artifacts were published.

    Source: §12.3 · p. 26

  • Takeaways & Limitations

    PEDAL makes documenting, versioning, and citing the exact instructions given to a model natural, treating prompts as the methodology of AI-mediated educational interventions.

    Source: §1 · p. 3

    The system has internal validation but lacks large-scale empirical studies comparing AI-generated scores with independent human reviewer assessments.

    Source: §11.3 · p. 24

Abstract

from arXiv · show

The rapid integration of Large Language Models (LLMs) into educational practice has created an urgent need for infrastructure that treats AI prompts not as disposable instructions but as reproducible scholarly artifacts. This paper presents PEDAL (Pedagogical Evaluation, Design, & Analysis Lab), an open-research platform implementing a three-tier Laboratory-to-Archive pipeline: (1) an Orchestration Layer with AI-assisted prompt generation and automated metadata extraction; (2) a Laboratory Layer supporting Git-style version control, LLM-as-a-Judge evaluation, and Mann-Whitney U statistical testing; and (3) a Public Archive Layer with per-version DOI minting via Zenodo, multi-format exports (JSON, CSV, LaTeX), and SEO-optimized discoverability. PEDAL's Scholarly Sync 2 (SS2) framework attaches a 24+ field metadata envelope encoding Bloom's Revised Taxonomy, Webb's Depth of Knowledge, SAMR levels, 5E phases, and NGSS alignment. A chemistry education exemplar demonstrates the full pipeline from Socratic inquiry scaffolding through statistical evaluation to DOI-minted archival. We further present NExAIE (Nexus AI & Education), applying PEDAL's infrastructure to AI-augmented peer review through a 42-prompt evaluation matrix spanning six quality dimensions, introducing Radical Transparency by publicly archiving all review rubrics with DOIs. Initial deployment data -- 4,684 views from 1,810 researchers within one month -- indicates strong demand for citable AI scaffolding in STEM education. Released under CC-BY-4.0 (DOI: 10.5281/zenodo.19474709).

1 Introduction

Educational prompts embody substantial pedagogical reasoning but remain ephemeral, while AI-assisted peer review faces opacity and transparency challenges. The paper introduces PEDAL’s citable prompt infrastructure and NExAIE’s extension to peer review.

1 Introduction

  • PEDAL treats prompts as scholarly research instruments, pairing version control, pedagogical metadata, DOI citation, and statistical comparison in an open Laboratory-to-Archive pipeline.Source: §1 · p. 2
  • NExAIE extends this infrastructure to peer review with a 42-prompt, six-dimension matrix, publicly DOI-archived rubrics, and privacy-preserving dual views.Source: §1 · pp. 2–3

2.1 Prompt Engineering as Scholarly Practice

  • Unlike prompt-sharing repositories that treat prompts as interchangeable code, PEDAL assigns each a DOI, version history, pedagogical metadata, and evaluation evidence.Source: §2.1 · p. 4

2.2 Open Science and the FAIR Principles

  • AI prompts depend on context and lack educational FAIR infrastructure; PEDAL makes them findable, accessible, interoperable, and reusable through public indexing, exports, DOI snapshots, and CC-BY licensing.Source: §2.2 · p. 4

2.3 The Crisis of Traditional Peer Review

  • Reported inter-reviewer agreement is κ=0.17–0.34, while interdisciplinary education journals struggle to find reviewers spanning needed expertise.Source: §2.3 · pp. 4–5

2.4 AI-Assisted Evaluation in Scholarly Publishing

  • AI-review approaches are criticized for opaque prompts, holistic rather than diagnostic scores, genre-blind criteria, and transparency–confidentiality trade-offs.Source: §2.4 · p. 5

2.5 Transparency and Reproducibility in Review

  • Because LLM review is determined by its instructions, NExAIE treats prompts as methodology requiring documentation, versioning, peer review, and public access.Source: §2.5 · pp. 5–6

3 PEDAL Platform Architecture

PEDAL separates prompt creation, evaluation, and publication in a Laboratory-to-Archive pipeline. Its architecture treats prompts as documented, versioned, evaluated, and archived scholarly artifacts.

3.1 Design Philosophy

  • PEDAL’s design treats prompts as laboratory instruments, makes pedagogical context structural metadata, and separates orchestration, evaluation, and publication into independent tiers.Artifacts can be refined through multiple evaluation cycles and published only after meeting explicit quality thresholds.Source: §3.1 · pp. 6–7

3 PEDAL Platform Architecture

  • PEDAL separates orchestration, laboratory evaluation, and public archiving, allowing artifacts to move through creation, refinement, and publication after meeting quality thresholds.Figure 1 presents the three-tier pipeline from prompt creation to DOI-minted publication, anchored by SS2 metadata.Source: Figure 1 · p. 6, §3.1 · p. 7

3.2 Layer 1: Orchestration

  • The Orchestration layer creates prompts through intent-aware generation, automated extraction of 24+ metadata fields, and PDF-based bootstrapping from educational materials.Its output is a draft Master Artifact with an initial version and populated metadata envelope.Source: §3.2 · p. 7

3.3 Layer 2: Laboratory

  • The Laboratory layer supports immutable Git-style branching and version history, LLM-as-a-Judge scoring across five dimensions, and execution of prompts with injected variables.Researchers currently execute prompts in external LLM interfaces and manually log results; the system returns aggregate scores and identifies the highest-scoring variant as Lab Preferred.Source: §3.3 · p. 9

3.4 Layer 3: Public Archive

  • The Public Archive displays metadata, version navigation, statistical summaries, and citations; it also provides per-version DOI citations, JSON, CSV, and LaTeX exports, and discoverability features.The archive defaults to the highest-scoring version and uses JSON-LD, HighWire metadata, and an XML sitemap to support indexing.Source: §3.4 · p. 11

3.5 Technology Stack Summary

  • Table 1 summarizes the PEDAL technology stack for version 1.5.0.The supplied passage identifies the table’s scope but does not specify its components.Source: Table 1 · p. 11

4 Scholarly Sync 2 and Research Classification

Scholarly Sync 2 (SS2) provides a machine-readable metadata envelope and protocol for prompt artifacts. PEDAL uses it to encode pedagogical classifications and distinguish artifact properties from version-specific properties.

4.1 The SS2 Metadata Framework

  • SS2 coordinates metadata for each prompt artifact, propagates status changes to public DOI records, logs sync events with timestamps and author UUIDs, and provides machine-readable exports.Exports include JSON research snapshots, CSV datasets, and LaTeX source code.Source: §4.1 · pp. 11–12

4.2 Cognitive Taxonomy Integration

  • Bloom’s Revised Taxonomy tags the highest cognitive level a prompt is designed to elicit, while Webb’s DOK classifies the complexity of thinking required.The two systems provide complementary classification dimensions.Source: §4.2 · p. 12

4.3 Pedagogical Framework Mapping

  • PEDAL encodes SAMR technology-integration levels and 5E inquiry-cycle phases to support archive filtering and curriculum-aligned prompt sequencing.SAMR spans Substitution through Redefinition; the 5E phases are Engage, Explore, Explain, Elaborate, and Evaluate.Source: §4.3 · p. 12

4.4 NGSS Standards Alignment

  • For science prompts, PEDAL links artifacts to NGSS standards using records for the standard code, description, SEP, CCC, and DCI, validated against a master registry.The registry is identified as the authoritative standards catalog.Source: §4.4 · p. 13

4.5 Dual-Track Classification

  • PEDAL stores invariant subject, audience, pedagogical-intent, textbook-alignment, and NGSS properties at prompt level, while version-level metadata tracks changing Bloom’s level, strategy, model, and system prompt.This distinction supports tracking how cognitive level or prompting approach changes across refinement cycles.Source: §4.5 · p. 13

5 Statistical Engine and DOI Integration

PEDAL uses Mann–Whitney U testing to compare prompt variants and a five-dimension rubric to score executions. The Lab Preferred badge requires both the highest average quality score and statistical significance against a baseline.

5.1 Mann–Whitney U Test Implementation

  • PEDAL uses the Mann–Whitney U rank-sum test for ordinal 1–5 scores, typically small samples (n < 30), and rarely normal score distributions.The implementation includes tie and variance corrections and computes a Z-score using a normal approximation.Source: §5.1 · pp. 13–14

5.2 LAB PREFERRED Criteria

  • A variant earns Lab Preferred only if it has the highest average quality score across the version lineage and is significant at p < 0.1 against the baseline version.The threshold is relaxed from the conventional 0.05 to accommodate small samples in iterative prompt engineering.Source: §5.2 · p. 14, §5.1 · p. 14

5.3 Rubric Scoring System

  • Each prompt execution is evaluated across five default rubric dimensions stored in the prompt_quality_rubric table, including Bloom’s Alignment, Socratic Integrity, Subject Precision, and Persona Consistency.Source: §5.3 · p. 14

5. Actionability (actionability): Can a practitioner implement the prompt output in a real classroom setting?

PEDAL documents rubric scores and uses Zenodo deposits to publish approved artifacts with version-specific DOIs. The deposit record supports updates, concept-level resolution, and author attribution.

5. Actionability (actionability): Can a practitioner implement the prompt output in a real classroom setting?

  • Each rubric dimension is scored from 1–5, and the aggregate quality score is the unweighted mean across five dimensions.Scores, rater identity and role, and free-text notes are stored per execution.Source: §5 · p. 14

5.4 Zenodo Integration Architecture

  • The Zenodo workflow creates a deposition or new version, then uploads the prompt text as a .txt file and a JSON metadata sidecar containing the SS2 envelope.These are the first three listed deposit steps.Source: §5.4 · p. 15
  • The deposit workflow sets creators, description, license, and keywords as Zenodo metadata before publication.The published deposit returns a version-specific DOI.Source: §5.4 · p. 15
  • PEDAL stores a version-specific DOI, a Zenodo deposition ID for updates, and a concept DOI that resolves to the latest version.These are stored as separate fields per artifact.Source: §5.4 · p. 15

5.5 Multi-Author DOI Support

  • Collaborative deposits list the first author as Creator 1 and the PI as Creator 2, with ORCID and affiliation included when available.Only artifacts with approved status can be deposited, so DOI minting follows the PI quality gate.Source: §5.5 · p. 15

6 Collaborative Access Control

PEDAL separates access between principal investigators and collaborators, then routes collaborative prompts through PI review before publication. Ordered author roles preserve attribution in citations and Zenodo deposits.

6.1 Role Architecture

  • The Principal Investigator has full platform access, while collaborators can edit their own prompts and run evaluations but cannot publish directly.Collaborators submit prompts for PI review.Source: §6.1 · p. 15

6.2 Approval Workflow

  • Prompts move from collaborator creation and submission to PI approval, rejection, or revision request; only approved prompts can reach production or Zenodo.Approval actions are logged with timestamps, reviewer identity, and PI notes.Source: §6.2 · p. 16

6.3 Multi-Author Attribution

  • Ordered author records assign first-author, PI, and co-author roles, and preserve author order in citation exports and Zenodo deposits.Source: §6.3 · p. 16

7 Implementation Exemplar 1: Chemistry Education Domain

The chemistry exemplar applies PEDAL’s Scholarly Sync 2 framework to solution chemistry, molarity, and dilution. It maps generated inquiry scaffolding into metadata and research-log fields, then archives the prompt artifact with a DOI.

7 Implementation Exemplar 1: Chemistry Education Domain

  • The exemplar applies Scholarly Sync 2 to solution chemistry, molarity, and dilution as a foundational chemistry education use case.Source: §7 · p. 16

7.1 Technical Mapping and SS2 Injection

  • Over 24 metadata fields are parsed from chemistry outputs to populate an SS2 envelope for LLM-generated inquiry-based scaffolding scenarios.Source: §7.1 · p. 16
  • The artifact is automatically mapped to Bloom’s Create level, DOK Level 4, and NGSS performance expectation HS-PS1-7 on conservation of atoms and mass.Source: §7.1 · p. 16

7.2 Research Scaffolding

  • Formal metadata frames the prompt as a citable research instrument, while its future_directions segment proposes research ideas, three questions, and three testable hypotheses.The section presents this scaffolding as suitable for academic inquiry, including a Master’s thesis.Source: §7.2 · p. 17
  • The platform’s research-log fields capture the scaffolding, and the complete prompt artifact, including system parameters and version history, is archived under a DOI.Source: §7.2 · p. 17, Table 4 · p. 17

8 Implementation Exemplar 2: The NExAIE Extension

NExAIE extends PEDAL’s Laboratory-to-Archive pipeline to AI-augmented peer review, while leaving manuscript management and formal decisions with OJS. Its review framework uses six dimension-specific evaluations and a synthesis step, with encrypted confidential data and citable protocols.

8 Implementation Exemplar 2: The NExAIE Extension

  • NExAIE is a PEDAL extension for AI-augmented peer review, inheriting the core platform’s versioning, metadata, and security protocols.Its comprehensive review methodology and empirical evaluation are addressed in a companion paper (Kahveci, 2026b).Source: §8 · p. 17

8.1 Design Principles

  • OJS retains manuscript submission, author-editor communication, and Accept/Reject decisions; PEDAL does not receive, store, or process submitted manuscripts.The Editor-in-Chief manually transfers a version-controlled prompt and structured JSON response, preserving human editorial judgment and model-agnosticism.Source: §8.1 · pp. 17–18

8.3 The 42-Prompt Evaluation Matrix

  • NExAIE assesses manuscripts with six independent, publication-type-specific prompts and structured evidence, calibrated from reviewer guidelines at four journals.The system boundary links an OJS article’s AI-Human Review Disclosure to the PEDAL protocol slug and version, while PEDAL stores the article DOI.Source: §8.3 · pp. 18–19, §8.2 · p. 18

8.4 Six Core Dimensions

  • All manuscripts are evaluated on a 1–5 scale across six dimensions; score 5 is calibrated as genuinely rare, occurring in <10% of reviews.Score 3 represents competent, publishable-with-revisions work.Source: §8.4 · p. 19

8.5 Publication Type Adaptation

  • The universal rubrics are adapted for seven publication types, with the adaptation matrix summarized in Table 6.Source: §8.5 · p. 19, Table 6 · p. 20

8.6 Prompt Architecture and JSON Schema

  • Each dimension prompt uses a standardized structure and returns JSON with a 1–5 score, confidence, evidence, recommendations, red flags, and a public note.The structure includes a publication-type-specific rubric and anti-inflation calibration directives.Source: §8.6 · p. 20

8.7 Synthesis Engine

  • A synthesis prompt combines all six dimension outputs into an overall assessment, scope fit, verdict recommendation, strengths and concerns, and revision guidance.Aggregate scores of at least 4.5 with no dimension below 3 suggest accept; two or more scores of 2 or lower suggest reject, and a 1 in merit or rigor triggers a reject advisory.Source: §8.7 · p. 20

8.8 Code and Security Integration

  • Confidential manuscript metadata is encrypted before storage, and variable substitution supports manuscript inputs while protecting researcher identities and feedback fields.Source: §8.8 · pp. 20–21

9 Radical Transparency and Dual-View Architecture

NExAIE publicly archives its evaluation prompts, rubrics, and scoring criteria as version-controlled, DOI-bearing PEDAL artifacts. Its public notes and dual-view design provide methodological context while limiting exposure of confidential review details.

9.1 The Radical Transparency Paradigm

  • NExAIE publishes all 42 dimension prompts and the synthesis prompt as version-controlled PEDAL artifacts with Zenodo DOIs, available for public browsing and pre-submission self-review.Source: §9.1 · p. 21
  • The public archive is intended to empower authors, make review rigor independently verifiable, and offer rubrics as citable instruments other journals may adopt or adapt.Source: §9.1 · p. 21

9.2 Socratic Public Notes

  • Public notes offer broad, forward-looking Socratic context alongside dimension scores without naming manuscripts, authors, or specific findings.Source: §9.2 · p. 21, §9.3 · p. 21

10 Data Model and Security

PEDAL extends its artifact infrastructure to NExAIE through schema additions, stable review identifiers, encrypted confidential metadata, and controls over access and LLM data handling.

10.1 Database Schema Extensions

  • NExAIE adds 13 columns to PEDAL’s prompts table for review records, while a separate sequence table maintains yearly counters for unique pseudocodes.The additions include review type and data fields, scores, verdicts, synthesis, encrypted metadata, article DOI, and share token.Source: §10.1 · pp. 21–22

10.2 Pseudocode System

  • NXR-YYYY-NNNN is a permanent public-facing identifier, URL slug, and citable reference for each Paper Review record.The sequence number is zero-padded; once assigned, the pseudocode is never changed or reused.Source: §10.2 · p. 22

10.3 Encryption at Rest

  • AES-256-CBC encryption protects confidential manuscript metadata at rest, and decryption occurs only in authenticated Editor View rendering.The IV is prepended to the ciphertext; the Public View rendering path never calls the decryption function.Source: §10.3 · p. 22

10.4 Share Token Access

  • Section Editors receive read-only access to the full Editor View through a 64-character cryptographic share token.Token validation uses constant-time comparison; tokens permit no write operations.Source: §10.4 · p. 22

10.5 LLM Data Privacy

  • The framework specifies Gemini Pro through Google AI Studio’s paid API tier, and PEDAL retains structured JSON evaluation output rather than complete manuscript texts.The passage states that paid-tier submissions are not used for training, retained beyond the session, or accessible to Google employees.Source: §10.5 · pp. 22–23

11 Discussion

The discussion positions PEDAL as FAIR-oriented infrastructure for pedagogical prompts and NExAIE as an extension for transparent AI-augmented review. It also identifies diagnostic, ethical, and empirical boundaries and outlines directions for further development.

11.1 Contributions to Open AI Research Infrastructure

  • PEDAL embeds pedagogical classification in prompt metadata, supporting taxonomy-filtered discovery, curriculum-aligned sequencing, and cross-study comparison.The discussion frames this as FAIR infrastructure for an artifact type not previously addressed by comparable infrastructure.Source: §11.1 · p. 23
  • NExAIE applies the infrastructure to transparent, reproducible AI-augmented peer review, linking review prompts, records, and published articles in one DOI-minted provenance chain.The authors contrast this provenance chain with traditional peer review.Source: §11.1 · p. 23

11.2 Implications for Scholarly Publishing

  • The 42-prompt matrix separates manuscript assessment into six dimensions, while public rubrics and Socratic public notes offer diagnostic guidance and selective disclosure.The example contrasts an Academic Merit score of 4 with a Methodological Rigor score of 2; detailed critique remains protected.Source: §11.2 · p. 23

11.3 Limitations

  • Inter-run consistency has not been formally validated, and explicit rubrics constrain but cannot eliminate training-data bias toward particular research traditions or methodological paradigms.The authors describe LLM scores as potentially variable when the same manuscript is evaluated twice with the same prompt.Source: §11.3 · p. 23
  • Human mediation limits throughput to the EiC’s availability, public prompts could enable gaming, and the current single-LLM implementation forgoes multi-model consensus.The authors say substance-focused rubrics and human editorial authority mitigate prompt gaming; multi-model consensus brings cost and complexity trade-offs.Source: §11.3 · pp. 23–24
  • Internal testing has not yet been complemented by large-scale studies comparing AI-generated scores with independent human reviewer assessments.This limitation applies to both PEDAL’s statistical engine and NExAIE’s review framework.Source: §11.3 · p. 24

11.4 Ethical Considerations

  • The ethical framework makes evaluation criteria public and citable, labels AI verdicts as suggested with editor override authority, minimizes stored data, and informs authors about AI review.The submission process makes the methodology publicly accessible before submission.Source: §11.4 · p. 24

11.5 Future Work

  • Future work includes a detailed NExAIE companion specification, reliability and human–AI agreement studies, multi-model evaluation, longitudinal prompt tracking, cross-journal adaptation, and API integration.The proposed API work would retain editorial oversight and the audit trail of the Copy & Log workflow.Source: §11.5 · p. 24

12 Conclusion

PEDAL treats pedagogical AI prompts as reproducible scholarly artifacts, and NExAIE applies the infrastructure to AI-augmented peer review. The conclusion reports early platform use and describes implications for STEM education and scholarly practice.

12.1 Summary of Contributions

  • PEDAL’s three-tier pipeline provides, to the authors’ knowledge, the first FAIR-oriented platform specifically for pedagogical AI prompt artifacts.It combines structured creation and metadata extraction, version control and statistical evaluation, and public archiving with per-version DOIs and multi-format exports.Source: §12.1 · p. 25
  • SS2 attaches 24+ metadata fields encoding Bloom’s Revised Taxonomy, Webb’s Depth of Knowledge, SAMR, 5E, and NGSS alignment to each artifact.The authors say this enables taxonomy-filtered discovery, curriculum-aligned sequencing, and cross-study comparison.Source: §12.1 · p. 25
  • NExAIE uses a 42-prompt matrix across six quality dimensions and seven publication types, with publicly archived rubrics and a privacy-preserving dual-view architecture.The prompts are calibrated against review criteria from Computers & Education, JRST, BJET, and IJSE.Source: §12.1 · p. 25

12.2 Implications for STEM Education and Research

  • The chemistry exemplar presents a Socratic solution-chemistry prompt whose artifact records instructional strategy, cognitive-taxonomy alignment, misconception targets, and execution protocols.The Future Research Directions module generates research ideas, questions, and testable hypotheses from disciplinary prompt content.Source: §12.2 · pp. 25–26
  • For educators, PEDAL supports documenting, sharing, and refining instructional prompts; for researchers, its versioned metadata and statistical engine support comparative study and secondary analyses.The conclusion also identifies machine-readable exports as usable in systematic reviews and meta-analytic frameworks.Source: §12.2 · p. 26

12.3 Evidence of Community Interest

  • 4,684 artifact views from 1,810 researchers were recorded about one month after the first pedagogical artifacts were published.The analytics were recorded as of May 6, 2026; the authors report no formal announcement beyond the platform’s discoverability mechanisms.Source: §12.3 · p. 26

12.4 Broader Impact and Vision

  • The authors frame public prompts as a reproducibility guarantee and metadata as pedagogical rationale, while NExAIE makes review criteria transparent, citable, and improvable.The conclusion states that the framework requires human mediation through Copy & Log, preserving editorial control and researcher authority.Source: §12.4 · p. 26

A Platform Interface Gallery

  • The platform and its NExAIE prompts, synthesis protocol, statistical engine, and public archive infrastructure are released under CC-BY-4.0 with DOI 10.5281/zenodo.19474709.The appendix states that the PEDAL Archive is publicly accessible and that platform specifications and version history are maintained under version control with Zenodo DOI provenance.Source: §12.4 · p. 26, A Platform Interface Gallery · p. 27
Loading 2610.00001v1…