Source-linked AI summary
Demonstration-Based Robotic Feeding Using Dynamic Movement Primitives
Kenzhi Iskandar Wong, Kitti Thamrongaphichartkul, Sri Harsha Turlapati, Yanshuo Lu, Kanichi Doi, Boyu Ma, Bin Zhou, Lin Yang, Ananda Sidarta, Lek Syn Lim, Arun-Kumar Kaliya-Perumal, Jianfei Yang, Domenico Campolo
TL;DR
Robot-assisted feeding must coordinate motion generation with food acquisition, utensil transport, and mouth localization, while demonstrated spoon motions need adaptation to the current scene. This paper combines DMPs, RGB-D mouth perception, AprilTag localization, and robot control to retarget and execute feeding motions. A real-world Kinova Gen3 experiment demonstrates feasibility, with motion limited by pre-execution pose estimates.
Problem
Transferring demonstrated spoon motions into a complete feeding pipeline that adapts to current scene locations remains a challenge in coordinating robot-assisted feeding tasks.
Source: §I · p. 1Method
The framework learns separate DMPs for demonstrated scooping and feeding motions, then retargets them using detected tray and mouth positions for sequential robot execution.
Source: §I · p. 1, C. Mouth Perception · p. 3, D. Task Execution Sequence · p. 4Results
A real-world Kinova Gen3 experiment demonstrated feasibility of executing the complete approach, scooping, and feeding sequence within the perceived scene.
Source: §VI · p. 5Takeaways & Limitations
DMPs preserve the demonstrated motion's overall shape and orientation evolution while allowing configurations to adapt to the detected tray pose and mouth position.
Source: §V · p. 5Because mouth and tray poses are estimated only before execution, the system cannot respond to later participant motion or mouth closure during feeding.
Source: §V · p. 5
Abstract
from arXiv · showhide
This paper presents a demonstration-based robotic feeding framework that integrates human demonstration, Dynamic Movement Primitive-based motion planning, vision-based scene perception, and robot control. A human operator first demonstrates the scooping and feeding motions while the position and Euler-angle orientation of the spoon are recorded using a motion-capture system. The recorded trajectory is segmented into scooping and feeding motions, which are encoded using separate Dynamic Movement Primitives. Before task execution, an RGB-D camera estimates the participant's mouth position, while two AprilTags are used for robot-frame registration and tray localization. The detected tray pose is used to position the scooping motion, whereas the detected mouth position defines the positional goal of the feeding motion. The approach, scooping, and feeding trajectories are then executed sequentially by a Kinova Gen3 manipulator through a ROS~2-based framework. A real-world experiment demonstrates the feasibility of the integrated pipeline and shows that the learned spoon motions can be reproduced within the detected experimental scene.
I. INTRODUCTION
The introduction motivates adapting demonstrated spoon motions to a complete feeding pipeline, then presents DMPs as a representation for encoding and retargeting those motions. DMPs shape stable nonlinear dynamics with a learned forcing term, while their initial and goal configurations and duration can be modified during reproduction.
I. INTRODUCTION
- A complete feeding pipeline must coordinate food acquisition, utensil transport, mouth localization, motion generation, and robot control.Robot-assisted feeding aims to support people with upper-limb motor impairments while reducing reliance on continuous caregiver assistance.Source: §I · p. 1
- Direct replay ties spoon motion to demonstration conditions, whereas this framework retargets learned scooping and feeding motions to detected tray and mouth locations.Continuous mouth tracking, online adaptation, food detection, and more complex food manipulation are outside the stated scope.Source: §I · p. 1
A. Dynamic Movement Primitives
- DMPs encode trajectories as stable nonlinear dynamical systems whose learned forcing terms shape demonstrated motions.Their initial and goal configurations and execution duration can be modified during reproduction.Source: A. Dynamic Movement Primitives · p. 2
- The paper represents end-effector configuration in six-dimensional Euclidean space using position and Euler-angle orientation.The Euler-angle vector contains roll, pitch, and yaw.Source: A. Dynamic Movement Primitives · p. 2
- The DMP transformation system uses current configuration, scaled generalized velocity, and goal configuration, with parameters controlling temporal scaling and convergence.A common choice βz = αz/4 yields critically damped nominal dynamics.Source: A. Dynamic Movement Primitives · p. 2
- A canonical phase variable decreases from one toward zero as the motion progresses, while the nonlinear forcing term shapes the generated trajectory.As the phase approaches zero, the forcing term vanishes and the system converges to its goal configuration.Source: A. Dynamic Movement Primitives · p. 2
B. DMP Learning
The DMP learning procedure derives a forcing term from the demonstrated trajectory and fits basis-function weights to it using regularized least squares.
B. DMP Learning
- The forcing term is learned from a demonstrated trajectory using the supervised DMP procedure, with the demonstration's initial and goal configurations and duration specified.The demonstrated velocity and acceleration are obtained from the recorded trajectory.Source: B. DMP Learning · p. 2
- For constant τ, substituting the demonstration into the transformation system yields a demonstration-derived forcing term.The learned forcing term is represented as a weighted combination of basis-function vectors.Source: B. DMP Learning · p. 2
- Regularized least-squares regression fits the basis-function weights to the demonstration-derived forcing term.The collected forcing terms and basis-function vectors span the recorded trajectory, and λ is the regularization coefficient.Source: B. DMP Learning · p. 2
A. System Overview
The ROS 2 framework separates demonstration and DMP learning from task execution. It combines learned motion parameters with pre-execution scene perception to generate robot motions.
A. System Overview
- The framework has two stages: human demonstration and DMP learning, followed by robotic task execution.The operator demonstrates scooping and feeding, which are segmented to extract DMP weights and initial and final poses.Source: A. System Overview · pp. 2–3
- Before execution, an RGB-D camera detects the mouth and two AprilTags support tray localization and camera-to-robot frame registration.The mouth position is measured once before movement and remains fixed during execution.Source: A. System Overview · p. 3
- The motion planner combines learned DMP weights, task-relevant configurations, tray pose, and mouth position to generate sequential motions for manipulator execution.The overview depicts the learned motion and scene inputs being used to generate desired end-effector motion.Source: A. System Overview · p. 3, Figure 1 · p. 3
B. Demonstration and Task Segmentation
A motion-capture demonstration records spoon position and Euler-angle orientation, then divides the task into scooping and feeding segments learned independently as DMPs. During execution, the detected scene adjusts selected task poses.
B. Demonstration and Task Segmentation
- Motion capture records the spoon's Cartesian position and Euler-angle orientation during the human demonstration.The demonstrated trajectory has total duration Td.Source: B. Demonstration and Task Segmentation · p. 3
- The demonstrated trajectory is divided at a transition instant into scooping and feeding segments, which are learned independently as DMPs.The post-scooping pose serves as both the scooping segment's final pose and the feeding segment's initial pose.Source: B. Demonstration and Task Segmentation · p. 3
- During execution, the final feeding pose's position is replaced with the detected mouth position while its demonstrated Euler-angle orientation is retained.The demonstrated poses are adjusted according to the detected tray pose and mouth position.Source: B. Demonstration and Task Segmentation · p. 3
C. Mouth Perception
Mouth position is estimated from RGB-D imagery and transformed into the robot base frame; the feeding goal retains the demonstrated spoon orientation because only mouth position is estimated.
C. Mouth Perception
- DECA estimates facial landmarks from RGB images, and mouth landmarks plus depth yield the mouth’s three-dimensional camera-frame position.The mouth center is determined from the landmarks; corresponding depth information supplies its 3D position.Source: C. Mouth Perception · p. 3
- The camera-to-robot transformation expresses the estimated mouth position in the robot base frame.The homogeneous transformation BTC maps the camera frame to the robot base frame.Source: C. Mouth Perception · p. 3
- The feeding DMP retains the demonstrated final spoon orientation while using the estimated mouth position as its positional goal.The mouth position is estimated once before movement and remains fixed during execution.Source: C. Mouth Perception · p. 3
D. Task Execution Sequence
The task proceeds through approach, scooping, and feeding, with tray-relative demonstration poses retargeted to the detected tray and the feeding DMP directed toward its goal.
D. Task Execution Sequence
- Execution has three consecutive phases—approach, scooping, and feeding—and begins by detecting the tray AprilTag and mouth position in the robot base frame.The approach moves from a predefined home configuration to the pre-scooping configuration.Source: D. Task Execution Sequence · pp. 3–4
- Demonstrated pre- and post-scooping positions are represented relative to the tray AprilTag frame and transformed using the detected tray pose.The resulting pre- and post-scooping configurations combine those positions with the demonstrated spoon orientations.Source: D. Task Execution Sequence · pp. 3–4
- The scooping DMP uses its learned weights to generate spoon motion from the pre-scooping to the post-scooping configuration.Source: D. Task Execution Sequence · p. 4
- The feeding DMP starts from the post-scooping configuration, and its learned trajectory is integrated with the approach and scooping intervals into the desired end-effector trajectory.The approach trajectory moves from home to the pre-scooping configuration.Source: D. Task Execution Sequence · p. 4
E. Robot Control
ROS 2 transmits the desired end-effector configuration to a Cartesian impedance controller, which tracks position and orientation with compliant behavior.
E. Robot Control
- The Cartesian impedance controller tracks desired spoon position and orientation while providing compliant end-effector behavior.The desired end-effector configuration is transmitted through ROS 2 as the controller’s reference trajectory.Source: E. Robot Control · p. 4
- The experimental setup pairs a spoon-equipped Kinova Gen3 manipulator with an RGB-D camera and two AprilTags for mouth estimation, frame registration, and tray localization.Source: Figure 2 · p. 4
IV. EXPERIMENT
The experiment combines a motion-captured human demonstration with scene perception and robot control. The robot then executes the planned spoon motions in the detected scene, with qualitative perception and trajectory results reported.
A. Experimental Setup
- A 7-DoF Kinova Gen3 with a spoon operates beside a tray and bowl; one AprilTag registers camera and robot frames and another localizes the tray.Source: A. Experimental Setup · p. 4
- One operator’s motion-captured spoon position and Euler-angle orientation were segmented into scooping and feeding demonstrations for DMP learning.Demonstrated pre- and post-scooping positions were represented relative to the tray frame for retargeting.Source: A. Experimental Setup · p. 4
- ROS 2 integrates perception, motion planning, and robot control; mouth and tray poses are detected before each trial and used to generate the three motions.Source: A. Experimental Setup · p. 4
B. Experimental Results
- RGB-D mouth landmarks and depth determine the mouth’s three-dimensional position, while AprilTags support robot-frame registration and tray localization for positioning the scoop poses and feeding goal.Figure 3 presents qualitative mouth-perception results under different head poses and mouth configurations.Source: B. Experimental Results · p. 5, Figure 3 · p. 5
- The robot approaches the bowl, scoops, transports the spoon to the participant, and reaches the initially detected mouth position; planned end-effector trajectories are shown over the scene point cloud.The plotted coordinate frames show spoon position and orientation throughout the task.Source: B. Experimental Results · p. 5
V. DISCUSSION
The discussion emphasizes that DMPs preserve the demonstrated motion’s shape and orientation while adapting initial and final configurations to detected tray and mouth poses. The conclusion reports a Kinova Gen3 experiment executing the complete task sequence in the perceived scene.
V. DISCUSSION
- DMPs preserve the demonstrated spoon motion’s overall shape and orientation while adapting its initial and final configurations to the detected tray pose and mouth position.Source: §V · p. 5
- Performance depends on RGB-D mouth localization and AprilTag frame registration, and pre-execution pose estimates cannot respond to later participant motion or mouth closure.The paper identifies continuous mouth perception, online motion adaptation, and spoon retraction when the mouth closes as future work.Source: §V · p. 5
VI. CONCLUSION
- A real-world Kinova Gen3 experiment demonstrated execution of the complete approach, scooping, and feeding sequence within the perceived scene.Source: §VI · p. 5