Source-linked AI summary
Designing for human-AI complementarity in K-12 education
Kenneth Holstein, Vincent Aleven
TL;DR
The paper addresses how to design and evaluate teacher–AI partnerships in AI-supported K–12 classrooms, where teachers’ intuitions about students’ needs may be limited. It presents participatory, iterative design and classroom evaluation of Lumilo; students learned more with teachers using the analytics glasses than in either comparison condition.
Problem
How teacher-support tools should be designed and tested in AI-tutor classrooms remains an open question, including whether teachers’ intuitions alone identify where attention is most needed.
Source: Iterative piloting: Measuring and… · p. 5Method
The team co-designed Lumilo with teachers and iteratively used Causal Alignment Analysis, classroom pilots, and observations to refine the tool and study teacher–AI decisions.
Source: Design and development of Lumilo · p. 3, Iterative piloting: Measuring and… · p. 5Results
Students learned more with Lumilo than in business-as-usual and glasses-only conditions, and its analytics provided benefit beyond the glasses alone.
Source: In-the-wild evaluation study · p. 8Takeaways & Limitations
Effective human–AI partnerships call for practitioner involvement, study in real-world contexts, and theory-building about complementarity.
Source: In-the-wild evaluation study · p. 10The evaluation compared business-as-usual with a stripped-down glasses condition designed to gauge monitoring novelty and isolate analytics effects.
Source: In-the-wild evaluation study · p. 7
Abstract
from arXiv · showhide
Recent work has explored how complementary strengths of humans and artificial intelligence (AI) systems might be productively combined. However, successful forms of human-AI partnership have rarely been demonstrated in real-world settings. We present the iterative design and evaluation of Lumilo, smart glasses that help teachers help their students in AI-supported classrooms by presenting real-time analytics about students' learning, metacognition, and behavior. Results from a field study conducted in K-12 classrooms indicate that students learn more when teachers and AI tutors work together during class. We discuss implications of this research for the design of human-AI partnerships. We argue for more participatory approaches to research and design in this area, in which practitioners and other stakeholders are deeply, meaningfully involved throughout the process. Furthermore, we advocate for theory-building and for principled approaches to the study of human-AI decision-making in real-world contexts.
Design and development of Lumilo
Teachers shaped Lumilo through participatory research and iterative prototyping, producing smart glasses that add real-time student information to teachers’ classroom view.
Design and development of Lumilo
- Teachers emphasized that classroom support should augment their existing attention to students’ body language and facial expressions, not distract from it.They described much of their current insight as “reading the classroom.”Source: Design and development of Lumilo · p. 3
- Replay Enactments let teachers experience algorithm-design consequences using immersive simulations based on actual classroom data.Researchers changed algorithmic elements live in response to stakeholder feedback, including model parameters.Source: Design and development of Lumilo · pp. 3–4
- The resulting mixed-reality glasses display real-time indicators above students to alert teachers to situations needing attention.Figure 1 shows a default class view with student indicators and class-level analytics, plus “Deepdive” screens opened by selecting an indicator.Source: Design and development of Lumilo · pp. 4–5, Figure 1 · p. 4
- On request, Lumilo presents diagnoses alongside recent student errors, helping teachers question AI judgments and infer underlying difficulties.The HoloLens heads-up display lets teachers keep their attention on the classroom while viewing this information.Source: Design and development of Lumilo · p. 5
Iterative piloting: Measuring and shaping human–AI decision-making
The team iteratively tested Lumilo in classrooms, using measures of teacher behavior and observations to refine the tool and examine teacher–AI complementarity.
Iterative piloting: Measuring and shaping human–AI decision-making
- Causal Alignment Analysis starts from desired outcomes, maps hypothesized paths through student and teacher behavior, and uses real-world data to revise both designs and hypotheses.The approach works backward from goals such as improving learning or engagement measures.Source: Iterative piloting: Measuring and… · p. 5
- Across four pilots involving 14 teachers, 15 classrooms, and 304 students, the team tracked teacher activity and refined Lumilo to direct attention where it was most needed.The assessment drew on classroom observation and causal modeling of how students learn with AI tutors.Source: Iterative piloting: Measuring and… · p. 6
- The pilots combined quantitative measures of teacher behavior with qualitative observations to develop an account of how the partnership might benefit classrooms.These observations helped the researchers understand how teacher and AI decisions played out under real-world conditions.Source: Iterative piloting: Measuring and… · p. 6
- Classroom observations showed teachers combining AI alerts with their own knowledge and sensory cues to interpret students’ situations and decide how to respond.In one case, a teacher checked in after an off-task alert and learned that a student was upset about a breakup.Source: Iterative piloting: Measuring and… · p. 6
- Teachers’ brief visits often complemented the tutor through reflection prompts and encouragement rather than direct math coaching.Visits averaged about 24 seconds per student, and teachers could revisit students during a class.Source: Iterative piloting: Measuring and… · p. 6
In-the-wild evaluation study
A classroom study compared Lumilo with business-as-usual and glasses without analytics, finding greater student learning with Lumilo and additional benefit from its analytics. The team also examined how teacher attention and student gains varied with students’ incoming knowledge.
In-the-wild evaluation study
- The glasses-only condition gauged possible monitoring novelty effects, while comparison with no glasses helped isolate the contribution of analytics.The glasses-only version allowed remote screen monitoring but provided no analytics.Source: In-the-wild evaluation study · p. 7
- The study involved 343 middle-school students in 18 classrooms across 4 schools, randomly assigned within teacher to three conditions.Students worked with an AI tutor on linear equation solving for two class sessions between pre- and post-tests; 286 students were included in analyses.Source: In-the-wild evaluation study · pp. 7–8
- Teachers using Lumilo had higher student learning than in both business-as-usual and glasses-only conditions, with real-time analytics contributing benefit beyond the glasses.Figure 2 displays student pre/post learning gains by condition; the authors describe this as the first experimental demonstration of learning benefits from a teacher–AI partnership facilitated by tutor analytics.Source: Figure 2 · p. 7, In-the-wild evaluation study · p. 8
- In the analytics condition, teachers spent more time with lower-pretest students, who gained more; high-pretest students were unaffected, attenuating “rich get richer” differences.Figures 3 and 4 show teacher attention and student posttest scores by pretest score across conditions.Source: Figure 3 · p. 8, In-the-wild evaluation study · p. 9, Figure 4 · p. 9
- The authors identify practitioner participation, real-world study of human–AI decisions, and theory-building about complementarity as ingredients for effective partnerships.They note that methods for engaging nontechnical stakeholders with AI design and further theory formation remain open research needs.Source: In-the-wild evaluation study · p. 10