PaperlayerView exact revision on arXiv

Source-linked AI summary

AIfred: Augmented Learning through Functional Robotic Embodiment at the Desk

Gregorio Orlando, Milan Groshev, Eduardo Castelló Ferrer

arXiv:2609.38737v1cs.ROcs.HC

TL;DR

Desk-based work takes place on paper, while generative-AI guidance is delivered on a separate screen. AIfred uses workspace perception and a robotic projector to place task-relevant AI guidance alongside physical work. In a 36-person study, AIfred improved short-term transfer and drawing results, while math scores were comparable when assistance was available.

  • Problem

    Desk-based learning and creative work happen on physical surfaces, but current AI tools deliver guidance through screens, separating assistance from the workspace.

    Source: §I · p. 1

  • Method

    AIfred combines workspace perception, context-aware AI content generation, and robot-mediated projection across math, image-generation, and drawing tasks.

    Source: §I · p. 1

  • Results

    Professors ranked AIfred drawings first in 33 of 36 cases, while short-term transfer was 2.6 points higher on a 10-point scale.

    Source: §VII · p. 8

  • Takeaways & Limitations

    Spatially co-located assistance is especially relevant for tasks requiring users to align visual guidance with their physical work.

    Source: §VI · p. 7

    The prototype relies on OptiTrack motion capture, which limits deployment beyond controlled lab settings.

    Source: A. Limitations and Future Work · p. 7

Abstract

from arXiv · show

Desk-based learning and creative activities benefit from handwritten engagement. However, current generative AI tools deliver guidance through a separate screen, creating a gap between where users think and where assistance appears. To address this, in this work we design AIfred, a desk-based robotic arm with a projector mounted at the end-effector that places AI-generated guidance alongside handwritten work. AIfred combines workspace perception, context-aware content generation, and robot-mediated projection to support math assignments, image generation, and drawing tasks. In a user study (n = 36), we compared AIfred against ChatGPT (GPT-5.6 Luna) running on a laptop. Both tools performed comparably while assistance was available during the math assignment (6.7 vs. 7.3/10, p = .41), but AIfred improved short-term learning transfer by 60% once assistance was withdrawn (7.0 vs. 4.4/10, p = .003). In addition, independent art and design professors ranked drawings produced with AIfred better in 33 of 36 cases. Our findings indicate that spatially co-located AI assistance benefits tasks whose guidance shares a spatial frame with the work.

I. INTRODUCTION

The introduction identifies a mismatch between desk-based work on paper and AI guidance delivered on screens. It presents AIfred as a spatially co-located alternative and reports benefits in short-term learning transfer and drawing quality.

I. INTRODUCTION

  • AIfred combines a desk-based robotic arm and end-effector projector with overhead-camera workspace perception to deliver AI assistance alongside physical work.A motion-capture system tracks an object users move to indicate where projected content should appear.Source: §I · p. 1, Figure 1 · p. 1
  • AIfred’s pipeline perceives workspace context, generates context-aware content, and projects that guidance beside the user’s task.Source: §I · p. 1
  • The study compares AIfred with laptop-based ChatGPT across math, image-generation, and drawing tasks, assessing task outcomes, short-term transfer, context switching, and user experience.Source: §I · p. 1
  • With assistance removed, AIfred scored 7.0/10 versus ChatGPT’s 4.4/10 (60% higher transfer, p = .003); professors ranked AIfred drawings best in 92% of cases.While assistance was available for the math assignment, scores were 6.7/10 with AIfred and 7.3/10 with ChatGPT (p = .41).Source: §I · p. 2

II. RELATED WORK

Related work spans educational robots, projection-based spatial interfaces, and robotic projection platforms. The authors position AIfred as combining task interpretation and adaptive AI guidance with projection embedded in ongoing desk work.

II. RELATED WORK

  • Educational robots commonly serve as tutors or companions, while prior work has also used robots for demonstrations, tangible programming, and other functional learning support.The cited literature describes robots providing feedback, explanations, social interaction, and physical demonstrations.Source: §II · p. 2
  • Prior educational robots typically stand beside users as tutors or companions, and prior projection work generally separates material perception from adaptive AI interpretation.Source: §II · p. 2
  • Projection-based spatial interfaces have kept digital information co-located with physical objects, while instructional projection systems often rely on predefined content or scripted tutorials.Source: §II · p. 2
  • LuminAR enables manual repositioning of projected content, whereas ELEGNT explores expressive robot movement; neither addresses real-time AI interpretation of desk tasks.Source: §II · p. 2

III. METHODOLOGY

The methodology integrates a projector-equipped robotic arm into a desk workspace with motion capture, a trackable object, and an overhead camera. These components support repositioning projected assistance in response to the user’s physical task.

III. METHODOLOGY

  • The prototype places a mini projector on a robotic arm’s end-effector, with the arm positioned at the desk edge to reach and orient toward the workspace.Source: §III · pp. 2–3, Figure 2 · p. 3
  • Motion capture tracks the robot base and a user-moved desk object; inverse kinematics aims the projector at the target while maintaining a fixed height above the desk.Source: §III · p. 3
  • An overhead camera captures task context and pointing gestures, which the software uses to infer the task and generate corresponding projected output.Source: §III · p. 3

A. Spatial assistance pipeline

AIfred’s spatial assistance pipeline uses workspace perception, context-aware content generation, and robot-mediated projection. The described interaction modes include a range of desk activities, with math homework designed to scaffold rather than replace problem solving.

A. Spatial assistance pipeline

  • Workspace perception captures desk materials and detects pointing gestures; a screenshot of the indicated area and the active mode guide AI content generation, then the tracked object sets projection location.The pipeline uses an overhead camera, MediaPipe gesture monitoring, and multimodal AI models for scene understanding, text reasoning, or image generation.Source: A. Spatial assistance pipeline · p. 3

B. Interaction Modes

  • The three interaction modes span AI-assisted reasoning, creative transformation, and drawing quality, while users remain seated and work with physical materials.Source: B. Interaction Modes · p. 3

1) Math homework mode:

  • In math homework mode, users point to an exercise and receive projected hints, formulas, intermediate reasoning cues, or analogous examples beside their work.The design goal is scaffolding rather than directly solving the task for the user.Source: 1) Math homework mode · p. 3

2) Generate image mode:

AIfred turns a physical sketch into a refined or digitally cleaned image projected back onto the desk. In draw mode, it projects visual references onto the paper so users can improve an initial drawing.

2) Generate image mode:

  • AIfred transforms a selected physical sketch into a refined or digitally cleaned image, then projects it back onto the desk and saves it.The interaction scenarios are organized into requesting assistance, content generation and robot projection, and task completion.Source: 2) Generate image mode · pp. 3–4, Figure 3 · p. 4

3) Draw mode:

  • In draw mode, AIfred interprets an initial drawing and projects reference images, tutorials, or videos onto the paper to guide improvement.The projected guidance lets users compare references with their drawing without shifting attention between the desk and a screen.Source: 3) Draw mode · p. 4

IV. USER STUDY

The user study compared AIfred with laptop-based ChatGPT across math, image-generation, and drawing activities. Its measures covered task performance, physical-digital switching, and participants’ reported experience.

IV. USER STUDY

  • The mixed-design study compared AIfred with laptop-based ChatGPT, selected as a screen-based baseline, across the same task categories.The tasks were a quadratic-equation assignment, rendering a physical sketch as a digital image, and drawing a chosen figure.Source: §IV · p. 4
  • Participants completed assisted math and image-generation tasks, a three-condition drawing crossover, and a later unassisted math problem to assess short-term transfer.The second math assignment followed the first by an average of 35 minutes.Source: §IV · pp. 4–5

A. Performance metrics

  • Measures included observed desk-screen attention shifts, task time and quality, transfer, and six post-study experience ratings.Drawing rankings came from three independent professors (Kendall’s W = .86); math solutions were graded by four anonymized, randomized agents using a fixed rubric, with a mean inter-agent standard deviation of 0.53 points.Source: A. Performance metrics · p. 5

A. Drawing Quality

AIfred-assisted drawings received the strongest rankings, and AIfred’s math-assignment advantage emerged on the later problem after assistance was removed.

A. Drawing Quality

  • 33 of 36 AIfred drawings ranked first, compared with 3 ChatGPT drawings and none without assistance; all pairwise orderings were significant.The three professors’ rankings also showed improvements in overall form, proportions, and line quality for AIfred drawings.Source: Figure 4 · p. 5, Figure 5 · p. 5, A. Drawing Quality · p. 6

B. Math Assignment and Short-Term Learning Transfer

C. Image Generation Task Performance

AIfred’s image-generation time varied with the number of revisions, while its use was associated with far fewer physical-digital context switches and longer completion time. The reported explanation for the added time was sustained engagement with physical materials.

C. Image Generation Task Performance

  • 167 seconds with AIfred versus 208 seconds with ChatGPT was the average image-generation time (p = .32); AIfred was faster for one to three modifications, ChatGPT at four.Pointing at the physical sketch avoids image capture and upload, but subsequent AIfred revisions require physically modifying and re-rendering it.Source: C. Image Generation Task Performance · p. 6

D. Physical-Digital Context-Switching Behavior

  • 1 physical-digital switch with AIfred versus 63 with ChatGPT represented a 98% reduction, while completion times were 349.5 versus 259.9 seconds (p = .02).Participants looked at the screen only once per task on average with AIfred; the additional time was attributed to sustained engagement with task materials.Source: Figure 8 · p. 6, D. Physical-Digital Context-Switching… · p. 7

E. User Experience

AIfred received higher ratings for perceived learning support, innovation, and satisfaction, while participants reported low disruption from context switching despite frequent observed switches. The discussion links projected guidance to active paper-based engagement and argues that spatial co-location is most useful when guidance and work share a spatial frame.

E. User Experience

  • AIfred scored 4.5/5 vs. 3.4/5 for perceived learning support, 4.5/5 vs. 3.5/5 for innovation, and 4.2/5 vs. 3.5/5 for satisfaction.These differences were statistically significant (p < .001, p < .001, and p = .04, respectively).Source: Figure 9 · p. 7, E. User Experience · p. 7
  • Participants rated cognitive demand 3.4/5 vs. 2.9/5, while perceived productivity was statistically indistinguishable at 4.1/5 vs. 4.0/5.The respective p-values were .03 and .61.Source: E. User Experience · p. 7

VI. DISCUSSION

  • Projected guidance requires users to reinterpret and reproduce information by hand, which the authors link to deeper task engagement.Participants using AIfred sustained engagement on paper, retained more practiced material, and produced drawings professors preferred in the large majority of cases.Source: §VI · p. 7
  • For drawing, projected assistance was preferred, whereas AIfred and ChatGPT performed comparably on quadratic equations while assistance was available (6.7/10 vs. 7.3/10, p = .41).The discussion positions robot-mediated projection as targeted assistance, not a general-purpose replacement for screens.Source: §VI · p. 7
  • ChatGPT users made 63 observed physical-digital switches versus 1 with AIfred, although perceived switching disruption was low in both groups (1.7/5 vs. 1.3/5).The authors interpret this mismatch as attentional fragmentation operating below subjective awareness.Source: §VI · p. 7, E. User Experience · p. 7

A. Limitations and Future Work

AIfred’s study reports stronger drawing preferences and short-term learning transfer, alongside substantially fewer observed physical-digital switches. The authors note that motion-capture tracking limits deployment beyond controlled lab settings and identify vision-based tracking as a possible replacement.

A. Limitations and Future Work

  • OptiTrack motion capture limits deployment outside controlled lab settings; the authors identify vision-based tracking as a possible replacement.They also suggest exploring expressive, movement-driven robot behavior because the current design prioritizes functional projection.Source: A. Limitations and Future Work · p. 7

VII. CONCLUSIONS

  • The prototype uses an end-effector projector on a desk-based robotic arm to deliver generative AI assistance in the physical workspace.Its three interaction modes are math homework, image generation, and drawing.Source: §VII · p. 7
  • Art and design professors ranked AIfred drawings first in 33 of 36 cases, and short-term learning transfer improved by 2.6 points on a 10-point scale.The study included 36 participants comparing AIfred with a screen-based AI assistant across desk-based tasks.Source: §VII · p. 8
  • Participants made 63 physical-digital context switches with the screen-based assistant and 1 with AIfred, yet rarely registered switching as disruptive.The authors suggest the cost of attentional fragmentation may be largely invisible.Source: §VII · p. 8
Loading 2609.38737v1…