Source-linked AI summary
From Misconceptions to Evidence: What Science Teachers Make Visible When Co-Designing Agentic Learning Apps
Nizam Kadir, Wei Ting Liow, Sumbul Khan, Lay Kee Ang
TL;DR
Science teaching depends on eliciting learner reasoning and interpreting evidence, yet AI-generated content alone does not capture those judgments. This study analyzes four teacher co-design artifacts and finds that all specify evidence-bearing interactions, while teacher authority and safeguards are explicit in only half. It proposes a five-question protocol for specifying these arrangements before tools are built or adopted.
Problem
The study asks how science teachers translate epistemic teaching work into agentic-application specifications, since useful tools must embody judgments about learner reasoning and evidence rather than merely reproduce topic vocabulary.
Source: §1 · pp. 1–2Method
The authors conduct a bounded qualitative cross-case analysis of four de-identified science-teacher workshop artifacts, examining what each makes explicit.
Source: §3.1 · p. 4Results
Takeaways & Limitations
A five-question protocol covering problem, learner interaction, evidence, teacher authority, and safeguard can help teachers articulate accountable human-AI arrangements before design or adoption.
Source: §8 · p. 10The artifacts describe prospective applications without implementation, classroom-use, or outcome data, so the study cannot establish usability or learning effects.
Source: §3.1 · p. 4, §7 · p. 9
Abstract
from arXiv · showhide
Science educators increasingly encounter AI tools that generate content, yet disciplinary teaching depends on eliciting learners' models, diagnosing misconceptions, interpreting evidence, and preserving professional judgment. This study asks how science teachers translate such epistemic work into specifications for agentic learning applications. It contributes to the conference theme, "Innovating Pedagogies, Inspiring Minds: Transforming Science Learning," and the Teachers' Professional Learning strand by examining app co-design as a form of pedagogical reasoning. We conducted a bounded qualitative cross-case analysis of four de-identified artifacts produced in a teacher professional-learning workshop: an experimental-design diagnostic, a Kinetic Particle Theory dialogue guide, a chemistry prior-knowledge checker, and a physics application/scaffolding tool. Each artifact was coded for the disciplinary problem, learner interaction, evidence made visible, teacher authority, and safeguard. All four connected a science-learning problem to an interaction and pedagogically interpretable evidence: misconceptions and gaps, explanations-in-progress, class-level readiness patterns, or investigation performance. However, only two made teacher control or evaluation explicit, and only two named a safeguard. The proposals therefore positioned AI less as an answer generator than as an elicitor, scaffold, and evidence-return mechanism, while leaving decision rights and protections unevenly specified. We argue that teacher professional learning should treat AI app ideation as epistemic specification work. A five-question design protocol--problem, learner interaction, evidence, teacher authority, and safeguard--can help teachers transform science-learning needs into accountable human-AI arrangements before building or adopting a tool.
1 Introduction
Science teaching requires attention to learners’ reasoning and evidence, not only content delivery. This study asks what teachers make visible—and leave underspecified—when co-designing agentic learning applications.
1 Introduction
- A useful science application must embody teachers’ judgments about misconceptions, learner explanations or investigations, evidence, and when feedback should stop short of supplying an answer.The introduction frames the challenge as more than reproducing a prompt’s topic vocabulary.Source: §1 · pp. 1–2
- Teacher co-design can surface classroom routines, disciplinary bottlenecks, usable evidence, and boundaries on automation before software is built.App ideation is presented as a way to externalize a provisional theory of teaching and learning.Source: §1 · p. 2
- The study examines how four science-specific workshop artifacts translate disciplinary problems into agentic interactions and specify pedagogical action, evidence, teacher authority, and safeguards.The paper describes an agentic learning application as delegating responsive actions while learners and teachers remain part of the activity.Source: §1 · p. 2
- All four artifacts organized AI around eliciting, scaffolding, or interpreting scientific reasoning, but teacher decision rights and safeguards were explicit in only half.The authors propose a five-question protocol and emphasize that the artifacts do not establish whether any application would work or improve learning.Source: §1 · p. 2
2 Conceptual Background
The conceptual background frames science learning as disciplinary reasoning and formative assessment as an evidence-to-action process. It positions co-design as a way to make human-AI roles and teacher judgment discussable.
2.1 Science learning as epistemic work
- Science learning involves developing models, planning investigations, analyzing data, constructing explanations, and arguing from evidence; a correct answer alone can miss this epistemic work.Misconceptions and incomplete models can reveal how learners coordinate concepts and representations.Source: §2.1 · pp. 2–3
2.2 Formative assessment as an evidence-to-action relation
- Formative assessment links eliciting and interpreting learner thinking to action, so an agent’s educational role depends on disciplinary questions, visible reasoning, and authority to interpret or override its inferences.Automation may widen opportunities to elicit evidence while obscuring how an inference was made or encouraging a premature answer.Source: §2.2 · p. 3
2.3 Teacher agency, co-design, and educational AI
- Teacher agency includes shaping goals and interpreting context, while co-design can specify learner interactions, automated actions, returned information, decision rights, and safeguards.The paper calls this an epistemic specification when it links a disciplinary problem to interaction, evidence, decision rights, and safeguards.Source: §2.3 · p. 3
3 Method
The study analyzes four de-identified science-teacher workshop artifacts as design expressions, not participant-level responses. A five-dimension matrix supports descriptive cross-case comparison of what each artifact makes explicit.
3.1 Design and corpus
- The corpus comprises four science-specific proposals: an experimental-design diagnostic, a Kinetic Particle Theory dialogue guide, a chemistry prior-knowledge checker, and a physics application/scaffolding tool.One artifact was a group framing and three were structured proposals; the analysis treats each as a design expression.Source: §3.1 · p. 4
- No implementation, classroom-use, or outcome data are included; the unit of analysis is the content made explicit in each artifact.The cases were selected for depth and analyzable interaction content, not to represent teachers generally.Source: §3.1 · p. 4
3.2 Analytic procedure
- The analytic matrix covers disciplinary problem, learner interaction, evidence, teacher authority, and safeguards.The dimensions concern what learners do, what traces are interpretable, explicit teacher rights, and stated boundaries such as privacy or human review.Source: §3.2 · p. 4
- A dimension counted only when stated or directly entailed; counts are descriptive rather than statistical estimates, and Table 1 clarifies that “not explicit” refers to the artifact.The analysis used within-case summaries followed by comparison for common relations and negative contrasts.Source: §3.2 · p. 4, Table 1 · p. 5
4 Findings
Across four artifacts, science-learning problems became proposed interactions that elicited, scaffolded, or interpreted learner reasoning. Evidence was specified in every case, while teacher authority and safeguards were explicit less consistently.
4.1 The cases began with epistemic bottlenecks, not content topics alone
- The cases targeted epistemic bottlenecks—experiment-design reasoning, particle-model explanations, chemistry prerequisites, and physics application—rather than content topics alone.They varied from diagnostic moments to dialogue and scaffolded performance sequences, so “agentic app” did not imply one interface genre.Source: §4.1 · p. 5
4.2 Proposed agency centered elicitation, scaffolding, and repair
- Proposed applications would diagnose, question, differentiate, scaffold, or give feedback; none centered on supplying a finished scientific answer.The cases envisioned short diagnostic cycles, multi-turn dialogue, or an extended scenario-to-investigation sequence.Source: §4.2 · pp. 5–6
4.3 Every artifact specified evidence with disciplinary meaning
- All four artifacts made disciplinary evidence visible, from diagnosed misconceptions and developing explanations to class-level readiness patterns and assessable investigation performance.Evidence differed in audience and level: learner reflection, teacher interpretation, class readiness, or performance evaluation.Source: §4.3 · p. 6
4.4 Authority and safeguards were unevenly specified
- Explicit teacher authority appeared in two artifacts and explicit safeguards in two, although all four specified interaction and evidence.S3 let teachers select diagnostic concepts; S4 reserved evaluation or grading for teachers and named human review, while S2 named privacy but not approval or editing rights.Source: §4.4 · p. 6, Table 2 · p. 6
- The preserved specifications make interaction and evidence more consistently legible than decision rights and protections, and privacy and teacher control are not interchangeable.The authors caution that omissions do not establish that teachers considered authority or safety unimportant.Source: §4.4 · pp. 6–7
5 Discussion
The cases frame co-design as epistemic specification: teachers connect disciplinary problems, learner reasoning, and evidence while making professional judgments visible. They also show that evidence requires interpretation and action, and that visibility without explicit governance leaves accountability gaps.
5.1 App co-design externalized science teachers’ epistemic judgments
- Co-design externalized teachers’ judgments about where understanding becomes visible and how responsive support might move it forward.Personalization was tied to disciplinary thinking—such as experimental-design gaps, invisible models, prerequisite misconceptions, or transfer—not primarily content matching.Source: §5.1 · p. 7
- Designing an AI-supported activity can prompt teachers to articulate which reasoning learners retain, what the system may infer, and what evidence merits professional attention.This positions co-design as professional learning focused on pedagogical commitments rather than prompt tricks or product demonstrations.Source: §5.1 · p. 7
5.2 Evidence should be specified as part of an instructional relation
- All four artifacts made evidence central, but its audience, grain, and temporal role varied; designers should link learner action to disciplinary interpretation and a next decision.Evidence can make data visible without making learning intelligible, and its pedagogical value and ethical burden emerge together.Source: §5.2 · p. 7
- Evidence becomes formative through use, so app specifications should connect evidence to an interpretive actor and next instructional move, especially when automation infers misconceptions or personalizes feedback.Misconception labels may prompt revision or intervention, but opaque inference or fixed classification may harm; video investigations raise privacy, access, criteria, and review-workload questions.Source: §5.2 · p. 7
5.3 Epistemic visibility without governance is incomplete
- Unevenly specified teacher authority and safeguards create an accountability gap; oversight should name decisions such as reviewing inferences, editing feedback, or grading evidence.Privacy was explicit in S2 and human review in S4; other safeguards were not explicit in this small corpus and are facilitation questions, not faults in implemented systems.Source: §5.3 · p. 8
- Teacher–AI shared control can begin during ideation by specifying what automation may do, what evidence it returns, and which judgments remain professional.This makes governance part of pedagogical design rather than a later compliance addendum.Source: §5.3 · p. 8
6 Implications for Science Teaching and Teacher Learning
The paper proposes a five-question specification protocol for making science-learning needs and human-AI responsibilities explicit. It frames the protocol as a professional-learning and design aid, not an evaluated intervention or evidence of student outcomes.
6 Implications for Science Teaching and Teacher Learning
- The five-question protocol makes the epistemic sequence visible through its first three questions and uses the final two to govern each transition.Figure 1 is a conceptual synthesis from the four-case analysis, not an evaluated intervention.Source: Figure 1 · p. 8
6.1 For teacher professional learning
- Facilitators can guide teachers through describing a science-learning problem, mapping evidence to action, and reviewing authority and safeguards with peers.A specification is ready for further design when another teacher can identify who decides, what is visible, and what boundary prevents unacceptable use.Source: §6.1 · p. 8
6.2 For science teaching and prospective student learning
- For science teaching, diagnostics should reveal reasoning, dialogue should support explanation revision, class summaries should trace to meaningful concepts, and performance evidence should use visible criteria.Source: §6.2 · p. 9
- The proposed designs are implications rather than measured student outcomes; whether they improve understanding requires implementation, classroom enactment, and outcome measures.A next study could compare specifications made with and without the protocol and examine whether implemented interactions preserve learner work and teacher decision rights.Source: §6.2 · p. 9
7 Limitations
The analysis is bounded by its small, single-setting artifact corpus and by what the written artifacts make explicit. Its findings do not establish how proposed applications would work in practice or affect learning.
7 Limitations
- The study analyzes four artifacts from one professional-learning setting, selected for science specificity and analyzable interaction content, so they cannot represent the full workshop or science teachers generally.One artifact was group-authored and the others were posted proposals; artifact counts are not participant counts, and preserved text omits spoken deliberation and revisions.Source: §7 · p. 9
- The analysis concerns what artifacts made explicit, not teachers’ private knowledge or values; omitted safeguards may reflect brevity, elicitation order, or tacit norms.The five-dimension coding framework may leave other readings—such as accessibility, assessment validity, or classroom feasibility—less visible.Source: §7 · p. 9
- Because the artifacts describe prospective applications, the study makes no claims about implementation fidelity, feasibility, usability, workload, student experience, or learning effects.The five-question protocol is a conceptual implication and has not been evaluated.Source: §7 · p. 9
8 Conclusion
Across four science-teacher artifacts, agentic applications were imagined as elicitors, scaffolds, and ways to make scientific reasoning visible. The conclusion calls for treating app ideation as epistemic specification while keeping teacher authority and safeguards explicit.
8 Conclusion
- The four artifacts organized applications around eliciting, scaffolding, diagnosing, and making scientific reasoning visible rather than generic content generation.Experimental design, particle models, prerequisite chemistry knowledge, and real-world physics became distinct evidence-to-action relations.Source: §8 · p. 9
- Specifications articulated learner interaction and evidence more consistently than teacher authority and safeguards.Source: §8 · p. 9
- Treating AI app ideation as epistemic specification can help teachers clarify what an application should make visible and what must remain under professional control before design decisions become software.The paper cautions that this articulation does not guarantee a good tool.Source: §8 · p. 10