PaperlayerView exact revision on arXiv

Source-linked AI summary

PAMoR: Parameterized Affective Motion Generation in Real Time for Humanoid Robots

Yan Pan, Lingfan Bao, Tianhu Peng, Chengxu Zhou

arXiv:2608.28213v1cs.RO

TL;DR

Humanoid robots’ movements communicate affect, but existing methods lack an explainable, continuously parameterized, editable control signal for real-time whole-body generation. PAMoR computes valence and arousal directly from robot kinematics and composes separate action and affect priors during latent diffusion. On a physical 29-DoF Unitree G1, generated motion follows commanded V-A coordinates, preserves the requested action better than joint conditioning, and emotion recognition reaches 0.38 of trials versus 0.44 for acted human bodies.

  • Problem

    Existing affective motion methods use reference clips or emotion words, neither providing a quantitative motion-level parameterization for humanoid robots.

  • Method

    PAMoR computes valence from postural expansion and arousal from movement energy, then composes separate action, valence, and arousal diffusion priors in a shared latent space.

  • Results

    0.38 of trials identified commanded emotions, exceeding both baselines and approaching the 0.44 recognition rate reported for acted human bodies.

  • Takeaways & Limitations

    Real-time humanoid motion can preserve the requested action while affect is continuously controlled and edited through robot-native V-A coordinates.

Abstract

from arXiv · show

People read a humanoid robot's motion in social settings not only for the action performed but for the affect conveyed. Motion carrying that affect has so far been generated for human avatars, where style is taken from a reference clip or an emotion word, neither of which can be quantitatively parameterized. We present PAMoR, which turns affect into a measured control parameter: a valence-arousal (V-A) coordinate computed natively on robot kinematics. It is obtained in closed form from postural expansion and movement energy, and these measurements serve directly as generation conditions, with no human annotation. An action prior and two affect priors, trained in a shared latent space, are composed at each denoising step: the action prior fixes what is performed, the affect priors modulate how. Whole-body motion rolls out autoregressively on a 29-DoF Unitree G1 in real time, with action and affect both editable. Generated motion tracks the commanded V-A over its full range while text-to-motion fidelity still matches text-only baselines. In a perceptual study, raters identify the commanded emotion on 0.38 of trials, above both baselines and approaching the 0.44 reported for acted human bodies.

I. Introduction

PAMoR addresses the lack of explainable, editable affect control for real-time humanoid motion by measuring affect on robot kinematics and composing separate action and affect controls.

  • Motivation: Humanoid motion conveys affect in social settings, creating a need for parameterized, real-time, editable control on humanoid platforms.The paper identifies explainability, editability during execution, and humanoid embodiment as requirements.
  • Research gap: Reference clips entangle affect with action, while emotion words rely on annotator consensus and provide no motion-level definition.The paper positions continuous valence–arousal coordinates as the missing explainable parameterization.
  • Approach: PAMoR computes valence and arousal from robot kinematics in closed form, labeling the training corpus without human annotation.The approach maps posture and movement cues to a continuous V-A coordinate.
  • Approach: Three separately trained diffusion priors let text determine the action while valence and arousal modulate its style during real-time rollout on a 29-DoF Unitree G1.All three conditions remain user-specified and editable during autoregressive generation.

A. Motion Diffusion for Humanoid Robots

Prior work established text-to-motion diffusion and some robot affect control, but continuous whole-body V-A-conditioned generation on humanoids remains insufficiently addressed.

  • Motion diffusion: Human motion diffusion progressed from raw-space denoising to latent-space generation and real-time autoregressive rollout, but outputs still require robot retargeting.Inference-time retargeting adds latency and can alter affect-bearing postures and speeds.
  • Control gap: A single network jointly conditioned on text and V-A entangles affective control with action identity, so changing affect can distort the requested action.The paper motivates separate conditional models to avoid this coupling.
  • Related work: Human-avatar methods specify affect through reference clips, emotion labels, or speech-driven expressive motion rather than continuous whole-body humanoid control.These approaches differ in whether affect comes from example, label, or speech context.
  • Related work: Robot studies explored V-A control through offline optimization, latent editing of fixed clips, or short expressions, leaving broader real-time whole-body generation open.The cited systems span Cassie, a vacuum robot, Blossom, and Pepper.
  • Foundations: Body-motion research links positive valence with open posture and arousal with movement energy, while prior recognizers target affect recognition from annotated human motion.PAMoR builds generation conditions from these kinematic relationships instead.

III. Method

PAMoR formulates next-primitive synthesis for the Unitree G1 using robot-native motion features and closed-form V-A labels derived from posture and movement energy.

  • A. Problem Formulation: Each robot frame is represented by root orientation, heading change, foot contacts, root displacement and height, joint angles, and joint deltas.These features form the robot’s native motion vector x_t.
  • A. Problem Formulation: The framework generates the next F-frame primitive from an action prompt, target (v, a) ∈ [-1, 1]^2, and H-frame motion history.Autoregressive rollout composes primitives into a full motion for the 29-DoF G1.
  • B. V-A Labeling: Valence and arousal are computed in closed form from 14 forward-kinematics keypoints, using posture for valence and movement energy for arousal.The resulting labels require no human annotation and remain interpretable.
  • B. V-A Labeling: Valence combines arm expansiveness, root height, and trunk tilt, with each cue z-normalized and equally weighted.The cues increase as the body opens, covering lateral, vertical, and sagittal expansion.
  • B. V-A Labeling: Arousal combines z-normalized keypoint speed and acceleration magnitudes as a measure of movement energy.The per-frame labels are later averaged over each motion primitive.
  • B. V-A Labeling: Winsorized rescaling maps both raw scores to [-1, 1] using their 10th and 90th training-frame percentiles.Primitive-level V-A conditions are formed by averaging the rescaled frame labels.

C. Composable Latent Diffusion

The generator performs diffusion in a learned motion latent space, using a frozen Motion VAE to connect denoised latent tokens with future motion frames.

  • Latent diffusion: PAMoR composes three conditional priors at each denoising step in a shared motion latent space.The priors correspond to text, valence, and arousal conditions.
  • Latent diffusion: A 9-layer Transformer Motion VAE compresses H history and F future frames into a 128-dimensional latent token and decodes future frames from that token and history.Diffusion therefore operates on learned latent tokens rather than raw motion features.

1) Autoencoder:

PAMoR uses three separately trained diffusion priors in a shared latent space, combining text-defined action with independently controllable valence and arousal during denoising.

  • Composable Conditional Priors: Three priors share the same noisy latent, timestep, and motion history while receiving separate action, valence, or arousal conditions.Each prior uses the same Transformer denoiser architecture but independently trained weights.
  • Composable Conditional Priors: The text prior learns what action to perform, while affect priors learn how valence and arousal alter motion style.Valence corresponds to postural expansion, whereas arousal corresponds to movement energy.
  • Composable Conditional Priors: Separate affect priors estimate style directions independently of the action, allowing affect changes while the text prior preserves the requested action.Each affect prior is trained on motion pairs labeled by a scalar measured from that motion primitive.
  • Composable Conditional Priors: At each denoising step, classifier-free guidance combines the three conditioned predictions with their unconditioned counterparts.The composed clean-latent prediction is repeatedly noised and denoised until the decoder produces a motion primitive.
  • Autoencoder: A frozen MVAE encodes motion into the shared latent space before the three priors are trained independently.The shared space allows their predictions to be summed during sampling.

3) Training Objectives:

Training freezes a shared MVAE and independently optimizes the three priors with clean-latent diffusion losses plus motion-consistency terms, using data expanded for stylistic variation and transitions.

  • Training Objectives: Each prior regresses the clean latent with a Huber diffusion loss over training latents, timesteps, and noise draws.The loss uses the prior-specific condition and is supplemented by forward-kinematic and temporal-difference terms on decoded motion.
  • Training Objectives: Conditions are dropped with probability 0.1 during training to provide the unconditioned predictions required for classifier-free guidance.The priors compose only at sampling time, so individual priors can be added or removed without retraining.
  • Data Construction: The training corpus combines AMASS, video-reconstructed, and teleoperated motions to provide repeated actions across different kinematic styles.The added sources include expressive everyday actions performed in several styles.
  • Data Construction: Segments are retargeted or recorded in Unitree G1 space, grouped into 15 action classes, and chained in threes to expose action transitions.Equal class sampling prevents frequent actions from crowding out rare ones.

IV. Experiments

Experiments evaluate motion quality, text alignment, V-A controllability, perceived emotion, and implementation settings on a held-out robot-motion corpus.

  • Evaluation Protocol: The evaluation measures motion quality and text alignment with FID, R-precision, MM-Distance, Diversity, and MultiModality.Action accuracy is additionally reported in the ablation to test whether the requested action survives.
  • Evaluation Protocol: The corpus contains 10,095 training and 2,406 held-out chains across 15 action classes, sampled at 20 Hz on the Unitree G1.These chains yield 30,285 training and 7,218 held-out segments.
  • Evaluation Protocol: Baselines receive text alone, whereas PAMoR additionally receives valence and arousal values uniformly sampled from [−0.8, 0.8].Metrics use 1,000 randomly drawn held-out segments with a fixed seed and 30 generations per action class for MultiModality.
  • Evaluation Protocol: Valence and arousal are each swept over seven evenly spaced values, producing 49 commanded combinations per action prompt and three seeds per combination.The user study reports Top-3, first-pick weighted kappa, and naturalness.
  • Evaluation Protocol: Weighted kappa scores first-choice agreement, while naturalness is the mean rating on a five-point robotic-to-human-like scale.Weighted kappa ranges from 1 at perfect agreement to 0 at chance.
  • Implementation: Inference uses 10 DDPM steps per primitive, generates 8 frames at a time from 2 history frames, and applies guidance weights (5, 2, 2).All networks use Transformer denoisers with hidden size 512.

3) Implementation:

PAMoR preserves text-to-motion quality while adding affective variation, tracks commanded V-A values across the full range, and avoids the action-style trade-off observed in SMooDi.

  • Text-to-Motion Evaluation: FID, R-precision, MM-Distance, and Diversity stay on par with text-only baselines, while MultiModality rises under V-A conditioning.The reported result indicates added style variation without measurable cost to motion quality.
  • Baselines: The comparison evaluates TextOp, ECHO, and SMooDi, with SMooDi limited to R-precision because its training data differ.All methods are evaluated on one fixed set of 1,000 held-out prompts, while SMooDi is comparable only on R-precision.
  • Affect-Conditioned Baseline: SMooDi’s R@1 falls to 0.166, compared with roughly 0.79 for the three methods trained on the PAMoR corpus.Its ablation reports style accuracy increasing from 0.202 to 0.724 while R@3 decreases from 0.630 to 0.571.
  • V-A Controllability: V-A tracking reaches rank correlation 0.95, with slopes of +1.10 for valence and +0.79 for arousal across the full commanded range.The response is monotone, and cross-axis correlations remain below 0.06.

3) V-A control:

The user study tests whether raters can identify robot emotions from clips judged against neutral versions of the same actions. It uses ranked emotion guesses, naturalness ratings, and comparisons against chance and baseline generators.

  • User-study design: Seven actions are crossed with eight emotions, with every trial presenting an emotional clip beside a neutral reference of the same action.The design evaluates affect added to an action rather than emotion inferred from action identity.
  • Compared methods: The study compares text prompts, reference clips, and V-A coordinates as three ways of specifying emotion.TextOp uses emotive prompts, SMooDi uses reference clips, and PAMoR uses circumplex values.
  • User-study design: Raters rank their top three emotion guesses from eight options and rate motion naturalness on a five-point scale.The interface places the judged clip left of a neutral same-action reference.
  • Evaluation measures: 0.125 is the Top-1 chance level, while 0.375 is the Top-3 chance level for eight response options.These are the chance references used in the perceptual study.
  • Study protocol: Twelve naive raters completed 70 assigned trials each, producing six independent judgements per clip without exclusions.The 140 clips were divided into two balanced sets of 70.

2) Results:

PAMoR’s perceptual results show directionally accurate affect recognition, competitive performance with acted human-body recognition, and better action preservation than joint conditioning. The study also identifies action-dependent difficulty and a trade-off that composable priors avoid.

  • Error structure: Misread emotions are circumplex neighbors at 1.48 wedges on average for PAMoR, compared with 2.29 expected under random answering.The baselines reach 2.20 and 2.15 wedges, close to the random figure.
  • Human comparison: 0.44 recognition for acted human emotional bodies compares with PAMoR’s 0.38-level performance, which approaches the human-body result.Across emotions, PAMoR recognition ranges from 0.21 for relaxed to 0.49 for excited.
  • Action-dependent difficulty: 0.85 of positive-command trials received a positive first pick, despite punching and kicking being read as aggressive before affect is added.Across punching and kicking trials, first picks split 49 positive to 47 negative.
  • Ablation study: More than 0.2 is lost on both text retrieval and action accuracy by the single-prior variant under V-A modulation.The composable design preserves the requested action while V-A modulates style.

1) Composable versus single-prior diffusion:

PAMoR’s composable latent diffusion improves affect control and motion smoothness over motion-space diffusion while preserving independently editable action and affect. The framework remains limited by uncalibrated affect weighting and open-loop execution on the physical robot.

  • Latent versus motion-space diffusion: Arousal on-axis correlation rises from 0.79 to 0.95, while joint jerk drops by approximately 40% with latent diffusion.Slopes also move closer to 1, indicating tighter agreement between measured and commanded V-A values; jerk is reported in rad/s^3.
  • Latent versus motion-space diffusion: The latent variant makes commanded V-A values more visible and the resulting motion smoother than the motion-space variant.Both variants use the same composable design and are evaluated on the same V-A sweep.
  • Limitations: Equal weighting of posture and movement cues is a convention rather than a calibrated mapping, so further experiments are needed.The grounding literature identifies which cues raise valence or arousal but not how much each cue should count.
  • Limitations: Open-loop whole-body control can let executed V-A drift from the command because controller tracking errors are neither fed back nor affect-aware.Whether feedback can maintain commanded affect on the real robot remains an open question.
  • Composable versus single-prior diffusion: PAMoR composes separate action, valence, and arousal priors so action and affect remain independently editable during autoregressive real-time rollout.The action prior controls what is performed, while affect priors modulate how it is performed.
Loading 2608.28213v1…