Source-linked AI summary
Context-Aware Generation and Modulation of Expressive Motion Behavior using Multimodal Foundation Models
Till Hielscher, Fabio Scaparro, Kai O. Arras
TL;DR
Expressive robot motion must be selected, timed, and modulated to fit interaction context, yet prior work focused on generating expressive body language rather than automating what motion to perform and when. This paper uses multimodal foundation models to sequence and modulate DMP gestures within a continuous, distilled control framework. In a user study, its motions were rated as better aligned than non-modulated or API-level motions, and the distilled model reduced latency by 81.5% on average.
Problem
The paper addresses how to represent, select, time, and vary robot body motion so interlocutors perceive it as aligned with interaction context, extending prior work focused on generating expressive body language.
Source: §1 · pp. 1–2Method
The system uses multimodal foundation models to select, sequence, and modulate DMP-encoded whole-body gestures within a continuous control framework, with knowledge distillation for lower inference times.
Source: §1 · p. 2, §3.3 · p. 5Results
Participants rated the system’s motions as better aligned with context than non-modulated or API-level motions, and the distilled model reduced system latency by 81.5% on average.
Source: §6 · p. 8Takeaways & Limitations
Qualitative experiments demonstrate handling of multimodal context inputs and adaptability across robots with different kinematics.
Source: §6 · p. 8The work does not synchronize multiple output modalities, though temporal grounding in the DMP phase variable could enable such coordination.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
Expressive robot motion positively impacts human-robot interaction by improving user engagement, likability, or task performance. We present a novel approach to automatically generate and modulate complex, expressive full-body gesture sequences from a multimodal (text, audio, video) context description. Based on a unified mathematical implementation of the Principles of Animation using Dynamic Movement Primitives, we use multimodal foundation models to generate such sequences along with parametric motion variations that are highly context-aware. This method extends the state of the art in terms of flexibility and generality by being interpretable, composable, and working across different robot morphologies. Moreover, we integrate the system into a continuous control framework and leverage knowledge distillation to learn a much smaller model, significantly improving token efficiency and system latency. Results from a user study with a human-like platform indicate that participants judged our system's motions to be better aligned with the interaction context than motions produced under a non-modulated or API-level condition. We also demonstrate how the system generates expressive motion for robots with different kinematics, showcasing its versatility. Paper webpage: https://gen-mod-expressive-motion-behavior.github.io/.
1 Introduction
Expressive body motion matters for socially effective robot communication, but it must be selected, timed, and modulated to fit the interaction context. This paper automates that process using multimodal context and a composable motion representation.
1 Introduction
- Figure 1 illustrates a robot responding to a knock with a pause, turn, and neutral greeting, while a reunion context could prompt an exaggerated greeting.The illustration shows how context can change the modulation of a gesture.Source: Figure 1 · p. 1
- DMPs provide a unified implementation of the Principles of Animation with low-dimensional, interpretable parameters for stylizing motion across kinematic configurations.They also support execution toward static or dynamic spatial goals.Source: §1 · p. 2
- The paper automates context-aware selection, timing, and modulation of expressive robot motion, extending prior work focused on how to generate expressive body language.Its inputs include multimodal descriptions of context; the method targets what motion to perform and when.Source: §1 · pp. 1–2
- The system composes and modulates Dynamic Movement Primitives (DMPs) into expressive gestures using multimodal context, including text, image, and audio.The contributions describe the system as robot-agnostic with respect to kinematics.Source: §1 · p. 2
- Knowledge distillation transfers foundation-model knowledge to a smaller model, reducing inference cost and system latency.The paper also reports a user study on a human-like robot and qualitative demonstrations across a manipulator and a four-legged robot.Source: §1 · p. 2
2 Related Work
Related work spans high-level selection and composition of expressive behaviors, low-level motion representations, and co-speech gesture timing. This work combines general context understanding with physically grounded, cross-morphology gesture sequencing and modulation, while not addressing synchronization of multiple output modalities.
2.1 Expressive Motion Behavior Generation
- Prior work distinguishes high-level planning, which selects and composes movements, from lower-level generation, which synthesizes, learns, or executes basic movements.Rule-based systems require extensive manual understanding and classification; later approaches automate behavioral decision systems using traces, Bayesian networks, and learning methods.Source: §2.1 · p. 2
- The method generates sequences of complex, whole-body gestures from a pre-recorded DMP library, aiming for cross-morphology context-aligned motion rather than actuator calls or morphology-specific systems.The library also provides a risk-management option: gestures can be excluded if they may be perceived as political, obscene, or intensive.Source: §2.1 · p. 2
- Trajectory shaping with splines or polynomials offers flexibility but requires laborious manual design, while basic DMPs support smooth execution but lack expressive-motion flexibility.Other approaches adjust motion through VAD or Principles of Animation latent spaces, or through reinforcement-learning reward weights.Source: §2.1 · pp. 2–3
- Compared with methods using latent-space grounding, limited modulation, or lacking dynamic consistency, this work combines context-aware expressive adaptation with physically grounded execution and modulation.It extends prior work [9] with fully automatic, online-adaptive modulation parameterization.Source: §2.1 · p. 3
2.2 Co-Speech Gestures and Motion
- Co-speech gesture research coordinates speech and gestures through timing, semantic, temporal, or spatial grounding, but this work does not synchronize multiple output modalities.The authors note that temporal grounding in the DMP phase variable could enable such coordination.Source: §2.2 · p. 3
3 Method
The method uses multimodal context and robot-specific information to generate whole-body gesture sequences, modulate their expressive style, and execute them continuously. It combines a skill-library-based generation pipeline with DMP-based modulation, timeline control, and a distilled multimodal model.
3 Method
- Text, video, audio, and real-time sensory inputs form the interaction context used to guide motion behavior.Real-time inputs can include spatial positions from a person tracker or object detector.Source: §3 · p. 3
3.1 Sequence Generation
- A multimodal foundation model proposes a context-appropriate action sequence, checks it against robot capabilities, and selects matching library skills.The resulting plan is a sequence of library-instantiated, joint-space whole-body trajectories.Source: §3.1 · pp. 3–4
3.2 Sequence Modulation
- Foundation models describe how each primitive should be performed, then rate relevant Principles of Animation to translate that style into modulation parameters.Descriptions are generated per primitive while remaining coherent with the intent of the overall sequence; ratings cover six principles.Source: §3.2 · p. 5
- The ratings are projected onto robot-specific parameter ranges and applied to DMP trajectories to create a modulated sequence.The mapping is linear for Anticipation and Follow Through, piecewise linear for Arcs, Exaggeration, and Timing, and binary for Slow In Slow Out.Source: §3.2 · p. 5
3.3 System Overview
- A timeline controller links DMP primitives for continuous motion and can react to new context by replacing queued or active motion.At control frequency, the system can update a DMP goal to track a person’s changing location during execution.Source: §3.3 · p. 5
- Knowledge distillation trains a smaller multimodal model from traces of the multi-step pipeline, reducing intermediate inference steps and token use.Robot-specific preprocessing encodes trajectories as DMPs and derives kinematic information from the robot’s URDF.Source: §3.3 · pp. 5–6
4 Experiments
The experiments evaluate perceived context alignment, comparison with a multi-step pipeline, inference time, multimodal inputs, and operation across different robot morphologies. The user study uses videos of a human-like robot, while additional experiments examine latency and demonstrations on multiple platforms.
Evaluation overview
- The evaluation includes a user study, inference-time measurements, and demonstrations of multimodal context handling and operation across different kinematic structures.The study compares context alignment against an API-level baseline and examines whether distillation reduces reaction times.Source: §4 · p. 6
User study setup
- Thirty-eight participants were recruited, and 30 completed the survey; they rated randomized videos of Pepper and an interaction partner.Participants were aged 21 to 84; eight did not finish and their partial questionnaires were excluded.Source: §4 · p. 6
Study – Experiment 1
- In Experiment 1, participants rated context alignment on a 7-point scale across Ours, Ours (non-mod), and API-level conditions.The hypotheses compare non-modulated sequences with API-level behavior and modulated sequences with non-modulated sequences.Source: §4 · p. 6
Study – Experiment 2
- Experiment 2 compares Ours with Ours (multi) using a bipolar preference slider, with a stated non-inferiority hypothesis for the distilled model.The slider ranges from -100 to 100, with zero indicating equal perceived alignment.Source: §4 · pp. 6–7
Experiments 3–5
- Experiment 3 measures inference times and resulting system latency for Ours and Ours (multi), timing generation over 100 runs per context; Experiments 4–5 demonstrate motion generation across three robots and multimodal contexts.Pepper, Go2, and Gen3 Lite have 17, 12, and 6 degrees of freedom, respectively; demonstrations include an audible knocking sound and video plus spoken language used directly rather than transcribed.Source: §4 · p. 7
5 Results and Discussion
Across the user study, context alignment differed significantly between generation conditions, and the distilled model was non-inferior in preference ratings while reducing reaction time. Qualitative examples show context-sensitive gestures across robot morphologies and multimodal inputs, while the discussion notes library-size differences and a stronger rating effect for generation condition than context alone.
5.1 Quantitative Results
- Ours differed significantly from Ours (non-mod), and Ours (non-mod) differed significantly from API-level in context-alignment ratings (both p<0.001).The generation-condition effect was significant (p<0.001); Figure 3 shows the alignment ratings.Source: §5.1 · p. 7, Figure 3 · p. 8
- With a 10% slider-range margin, Ours was non-inferior to Ours (multi) in all three contexts; rating means favored Ours by 3.86%, 16.73%, and 7.56%, respectively.Evidence was strong for Context 1 (p<0.01) and very strong for Contexts 2 and 3 (p<0.001).Source: §5.1 · p. 7
- The distilled model averaged 2.39 s per context versus 12.87 s for Ours (multi), a 5.4× speedup and 81.5% lower reaction time.The models averaged 2161 and 4969 tokens, respectively; Figure 3 presents the latency comparison.Source: §5.1 · p. 7, Figure 3 · p. 8
5.2 Qualitative Results
- For Context 2, Pepper, Go2, and Gen3 Lite produced enthusiastic greeting gestures directed toward the approaching person, through gesture selection or modulation.Figure 4 visualizes the three robots’ motion behavior.Source: §5.2 · p. 7, Figure 4 · p. 8
- For Context 4, audio knocking changed the sequence to greeting and inviting the person closer; matching image and audio inputs for Context 2 preserved the text-conditioned motion.Without audio in Context 4, the method produced explaining and elaborating gestures instead.Source: §5.2 · p. 7
5.3 Discussion
- Generation condition had a larger rating effect (η²=0.447) than context (η²=0.0918); smaller gesture libraries on Go2 and Gen3 Lite coincided with repeated patterns.Pepper’s library contained over 30 basic gestures and produced diverse sequences.Source: §5.3 · pp. 7–8
6 Conclusion
The paper presents a system that sequences and modulates DMP-encoded whole-body gestures from multimodal interaction contexts. In a user study, its motions were rated as better aligned than non-modulated or API-level motions, while the distilled model reduced latency by 81.5% on average.
6 Conclusion
- The system uses multimodal foundation models to sequence and modulate DMP-encoded whole-body gestures from interaction context.It integrates these components into a timeline-based continuous controller and uses knowledge distillation.Source: §6 · p. 8
- Participants rated the system’s motions as better aligned with context than non-modulated or API-level motions; the distilled model showed no perceived context-alignment difference.The distilled model’s average system latency was reduced by 81.5%.Source: §6 · p. 8
- Qualitative experiments showed adaptation to different robot kinematics and multimodal inputs, including images, text, and sounds.The authors identify episodic memory, theory-of-mind models, and synchronized verbal output as possible extensions.Source: §6 · p. 8